{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Import Useful Data Libraries" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Import the CSV file – NSMES1988.csv into a dataframe" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0visitsnvisitsovisitsnovisitsemergencyhospitalhealthchronicadlregionagegendermarriedschoolincomeemployedinsurancemedicaid
01500001average2normalother6.9maleyes62.881000yesyesno
12102020average2normalother7.4femaleyes102.747800noyesno
231300033poor4limitedother6.6femaleno100.653200nonoyes
341605011poor2limitedother7.6maleyes30.658800noyesno
45300000average2limitedother7.9femaleyes60.658800noyesno
............................................................
440144021100000average0normalother8.4femaleyes82.249700noyesno
440244031200000average2normalother7.8femaleno115.813200noyesno
4403440410020011average5normalother7.3maleyes123.877916noyesno
440444051610000average0normalother6.6femaleyes123.877916noyesno
44054406000000excellent0normalother7.1maleyes06.596800yesnono
\n", "

4406 rows × 19 columns

\n", "
" ], "text/plain": [ " Unnamed: 0 visits nvisits ovisits novisits emergency hospital \\\n", "0 1 5 0 0 0 0 1 \n", "1 2 1 0 2 0 2 0 \n", "2 3 13 0 0 0 3 3 \n", "3 4 16 0 5 0 1 1 \n", "4 5 3 0 0 0 0 0 \n", "... ... ... ... ... ... ... ... \n", "4401 4402 11 0 0 0 0 0 \n", "4402 4403 12 0 0 0 0 0 \n", "4403 4404 10 0 20 0 1 1 \n", "4404 4405 16 1 0 0 0 0 \n", "4405 4406 0 0 0 0 0 0 \n", "\n", " health chronic adl region age gender married school \\\n", "0 average 2 normal other 6.9 male yes 6 \n", "1 average 2 normal other 7.4 female yes 10 \n", "2 poor 4 limited other 6.6 female no 10 \n", "3 poor 2 limited other 7.6 male yes 3 \n", "4 average 2 limited other 7.9 female yes 6 \n", "... ... ... ... ... ... ... ... ... \n", "4401 average 0 normal other 8.4 female yes 8 \n", "4402 average 2 normal other 7.8 female no 11 \n", "4403 average 5 normal other 7.3 male yes 12 \n", "4404 average 0 normal other 6.6 female yes 12 \n", "4405 excellent 0 normal other 7.1 male yes 0 \n", "\n", " income employed insurance medicaid \n", "0 2.881000 yes yes no \n", "1 2.747800 no yes no \n", "2 0.653200 no no yes \n", "3 0.658800 no yes no \n", "4 0.658800 no yes no \n", "... ... ... ... ... \n", "4401 2.249700 no yes no \n", "4402 5.813200 no yes no \n", "4403 3.877916 no yes no \n", "4404 3.877916 no yes no \n", "4405 6.596800 yes no no \n", "\n", "[4406 rows x 19 columns]" ] }, "execution_count": 95, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Load the CSV file into a DataFrame\n", "url = 'https://gperdrizet.github.io/FSA_devops/assets/data/unit2/NSMES1988-NSMES1988.csv'\n", "df = pd.read_csv(url)\n", "# Display the DataFrame\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Inital Obervations: \n", "1. The data was loaded from ''NSMES1988-NSMES1988.csv''\n", "2. Then I used JupyterNotebook's \"Pretty printout\" to visualize the dataframe.\n", "3. I initiall notice that there is a collumn named: 'Unnamed: 0' this is likely due to how to data was previously saved. \n", "4. Removing said column might be a good idea. \n", "5. income and age are encoded somehow. I need to contact whoever gave me the data to ensure that I properly handle these columns!!!!\n", "6. ..." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Inspect the data and report the details" ] }, { "cell_type": "code", "execution_count": 83, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0visitsnvisitsovisitsnovisitsemergencyhospitalhealthchronicadlregionagegendermarriedschoolincomeemployedinsurancemedicaid
01500001average2normalother6.9maleyes62.8810yesyesno
12102020average2normalother7.4femaleyes102.7478noyesno
231300033poor4limitedother6.6femaleno100.6532nonoyes
341605011poor2limitedother7.6maleyes30.6588noyesno
45300000average2limitedother7.9femaleyes60.6588noyesno
\n", "
" ], "text/plain": [ " Unnamed: 0 visits nvisits ovisits novisits emergency hospital \\\n", "0 1 5 0 0 0 0 1 \n", "1 2 1 0 2 0 2 0 \n", "2 3 13 0 0 0 3 3 \n", "3 4 16 0 5 0 1 1 \n", "4 5 3 0 0 0 0 0 \n", "\n", " health chronic adl region age gender married school income \\\n", "0 average 2 normal other 6.9 male yes 6 2.8810 \n", "1 average 2 normal other 7.4 female yes 10 2.7478 \n", "2 poor 4 limited other 6.6 female no 10 0.6532 \n", "3 poor 2 limited other 7.6 male yes 3 0.6588 \n", "4 average 2 limited other 7.9 female yes 6 0.6588 \n", "\n", " employed insurance medicaid \n", "0 yes yes no \n", "1 no yes no \n", "2 no no yes \n", "3 no yes no \n", "4 no yes no " ] }, "execution_count": 83, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Show the first few rows of the DataFrame\n", "df.head()" ] }, { "cell_type": "code", "execution_count": 84, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0visitsnvisitsovisitsnovisitsemergencyhospitalhealthchronicadlregionagegendermarriedschoolincomeemployedinsurancemedicaid
440144021100000average0normalother8.4femaleyes82.249700noyesno
440244031200000average2normalother7.8femaleno115.813200noyesno
4403440410020011average5normalother7.3maleyes123.877916noyesno
440444051610000average0normalother6.6femaleyes123.877916noyesno
44054406000000excellent0normalother7.1maleyes06.596800yesnono
\n", "
" ], "text/plain": [ " Unnamed: 0 visits nvisits ovisits novisits emergency hospital \\\n", "4401 4402 11 0 0 0 0 0 \n", "4402 4403 12 0 0 0 0 0 \n", "4403 4404 10 0 20 0 1 1 \n", "4404 4405 16 1 0 0 0 0 \n", "4405 4406 0 0 0 0 0 0 \n", "\n", " health chronic adl region age gender married school \\\n", "4401 average 0 normal other 8.4 female yes 8 \n", "4402 average 2 normal other 7.8 female no 11 \n", "4403 average 5 normal other 7.3 male yes 12 \n", "4404 average 0 normal other 6.6 female yes 12 \n", "4405 excellent 0 normal other 7.1 male yes 0 \n", "\n", " income employed insurance medicaid \n", "4401 2.249700 no yes no \n", "4402 5.813200 no yes no \n", "4403 3.877916 no yes no \n", "4404 3.877916 no yes no \n", "4405 6.596800 yes no no " ] }, "execution_count": 84, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Show the last few rows of the DataFrame\n", "df.tail()" ] }, { "cell_type": "code", "execution_count": 85, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0visitsnvisitsovisitsnovisitsemergencyhospitalhealthchronicadlregionagegendermarriedschoolincomeemployedinsurancemedicaid
12321233200011average4limitedother8.5maleyes80.9444nonono
10201021100000average1limitedmidwest8.3maleno90.8640nonono
988989300000average1normalwest7.2femaleyes81.8485noyesno
32243225300000average0normalmidwest9.2femaleno84.5448nonono
136137000000average2normalother8.0femaleno120.8400noyesno
14751476100000average1normalmidwest7.4femaleyes122.6086noyesno
22042205106000average4normalnortheast7.3femaleno80.5183nonoyes
296829692110001poor1limitedmidwest6.6femaleno80.4632nonono
25332534600000poor3normalmidwest7.8femaleno120.5984noyesno
422442253140000average1normalother8.1maleno40.4416nonono
\n", "
" ], "text/plain": [ " Unnamed: 0 visits nvisits ovisits novisits emergency hospital \\\n", "1232 1233 2 0 0 0 1 1 \n", "1020 1021 1 0 0 0 0 0 \n", "988 989 3 0 0 0 0 0 \n", "3224 3225 3 0 0 0 0 0 \n", "136 137 0 0 0 0 0 0 \n", "1475 1476 1 0 0 0 0 0 \n", "2204 2205 1 0 6 0 0 0 \n", "2968 2969 21 1 0 0 0 1 \n", "2533 2534 6 0 0 0 0 0 \n", "4224 4225 3 14 0 0 0 0 \n", "\n", " health chronic adl region age gender married school \\\n", "1232 average 4 limited other 8.5 male yes 8 \n", "1020 average 1 limited midwest 8.3 male no 9 \n", "988 average 1 normal west 7.2 female yes 8 \n", "3224 average 0 normal midwest 9.2 female no 8 \n", "136 average 2 normal other 8.0 female no 12 \n", "1475 average 1 normal midwest 7.4 female yes 12 \n", "2204 average 4 normal northeast 7.3 female no 8 \n", "2968 poor 1 limited midwest 6.6 female no 8 \n", "2533 poor 3 normal midwest 7.8 female no 12 \n", "4224 average 1 normal other 8.1 male no 4 \n", "\n", " income employed insurance medicaid \n", "1232 0.9444 no no no \n", "1020 0.8640 no no no \n", "988 1.8485 no yes no \n", "3224 4.5448 no no no \n", "136 0.8400 no yes no \n", "1475 2.6086 no yes no \n", "2204 0.5183 no no yes \n", "2968 0.4632 no no no \n", "2533 0.5984 no yes no \n", "4224 0.4416 no no no " ] }, "execution_count": 85, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Randomly sample 10 rows from the DataFrame\n", "df.sample(n=10, random_state=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Find out if the data is clean or if the data has missing values\n", "1. There seems to be no NaN values. However, that doesn't necessarilly mean there is no missing data. \n", "2. I should verify there is no missing data by ..." ] }, { "cell_type": "code", "execution_count": 86, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Unnamed: 0 0\n", "visits 0\n", "nvisits 0\n", "ovisits 0\n", "novisits 0\n", "emergency 0\n", "hospital 0\n", "health 0\n", "chronic 0\n", "adl 0\n", "region 0\n", "age 0\n", "gender 0\n", "married 0\n", "school 0\n", "income 0\n", "employed 0\n", "insurance 0\n", "medicaid 0\n", "dtype: int64" ] }, "execution_count": 86, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Count of missing values in each column\n", "df.isnull().sum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Comment on the data types, their values and range, specifically on age and income columns" ] }, { "cell_type": "code", "execution_count": 87, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 4406 entries, 0 to 4405\n", "Data columns (total 19 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 Unnamed: 0 4406 non-null int64 \n", " 1 visits 4406 non-null int64 \n", " 2 nvisits 4406 non-null int64 \n", " 3 ovisits 4406 non-null int64 \n", " 4 novisits 4406 non-null int64 \n", " 5 emergency 4406 non-null int64 \n", " 6 hospital 4406 non-null int64 \n", " 7 health 4406 non-null object \n", " 8 chronic 4406 non-null int64 \n", " 9 adl 4406 non-null object \n", " 10 region 4406 non-null object \n", " 11 age 4406 non-null float64\n", " 12 gender 4406 non-null object \n", " 13 married 4406 non-null object \n", " 14 school 4406 non-null int64 \n", " 15 income 4406 non-null float64\n", " 16 employed 4406 non-null object \n", " 17 insurance 4406 non-null object \n", " 18 medicaid 4406 non-null object \n", "dtypes: float64(2), int64(9), object(8)\n", "memory usage: 654.1+ KB\n" ] } ], "source": [ "# Display the columns, data types, and non-null counts of the DataFrame\n", "df.info()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data types: \n", " 0. Unnamed: 0 int64 --> unsigned int16 or int32 would be better if we keep this column. We don't expect 9 quintillion+ customers\n", " 1. visits int64 --> int8 or int16 Same logic can be applied as above, \n", " 2. nvisits int64 \n", " 3. ovisits int64 \n", " 4. novisits int64 \n", " 5. emergency int64 \n", " 6. hospital int64 \n", " 7. health object --> This is optimal for the current way health is encoded. \n", " 8. chronic int64 \n", " 9. adl object \n", " 10. region object \n", " 11. age float64 --> wouuld be better to use int8 or uint8 or if we need more precission on the current portion of a year, then float16\n", " 12. gender object \n", " 13. married object \n", " 14. school int64 \n", " 15. income float64 --> For the current encoding this might be optimal, but could be better if it wasn't encoded in a weird way. I want to have this encoded as an int32 as decimal precission doesn't matter (dependednt on method of encoding of course)\n", " 16. employed object \n", " 17. insurance object \n", " 18. medicaid object " ] }, { "cell_type": "code", "execution_count": 97, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countmeanstdmin25%50%75%max
Unnamed: 04406.02203.5000001272.0469721.00001102.250002203.500003304.750004406.0000
visits4406.05.7743996.7592250.00001.000004.000008.0000089.0000
nvisits4406.01.6180215.3170560.00000.000000.000001.00000104.0000
ovisits4406.00.7507943.6527590.00000.000000.000000.00000141.0000
novisits4406.00.5360873.8795060.00000.000000.000000.00000155.0000
emergency4406.00.2635040.7036590.00000.000000.000000.0000012.0000
hospital4406.00.2959600.7463980.00000.000000.000000.000008.0000
chronic4406.01.5419881.3496320.00001.000001.000002.000008.0000
age4406.07.4024060.6334056.60006.900007.300007.8000010.9000
school4406.010.2902863.7387360.00008.0000011.0000012.0000018.0000
income4406.02.5271322.924648-1.01250.912151.698153.1728554.8351
\n", "
" ], "text/plain": [ " count mean std min 25% 50% \\\n", "Unnamed: 0 4406.0 2203.500000 1272.046972 1.0000 1102.25000 2203.50000 \n", "visits 4406.0 5.774399 6.759225 0.0000 1.00000 4.00000 \n", "nvisits 4406.0 1.618021 5.317056 0.0000 0.00000 0.00000 \n", "ovisits 4406.0 0.750794 3.652759 0.0000 0.00000 0.00000 \n", "novisits 4406.0 0.536087 3.879506 0.0000 0.00000 0.00000 \n", "emergency 4406.0 0.263504 0.703659 0.0000 0.00000 0.00000 \n", "hospital 4406.0 0.295960 0.746398 0.0000 0.00000 0.00000 \n", "chronic 4406.0 1.541988 1.349632 0.0000 1.00000 1.00000 \n", "age 4406.0 7.402406 0.633405 6.6000 6.90000 7.30000 \n", "school 4406.0 10.290286 3.738736 0.0000 8.00000 11.00000 \n", "income 4406.0 2.527132 2.924648 -1.0125 0.91215 1.69815 \n", "\n", " 75% max \n", "Unnamed: 0 3304.75000 4406.0000 \n", "visits 8.00000 89.0000 \n", "nvisits 1.00000 104.0000 \n", "ovisits 0.00000 141.0000 \n", "novisits 0.00000 155.0000 \n", "emergency 0.00000 12.0000 \n", "hospital 0.00000 8.0000 \n", "chronic 2.00000 8.0000 \n", "age 7.80000 10.9000 \n", "school 12.00000 18.0000 \n", "income 3.17285 54.8351 " ] }, "execution_count": 97, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# General statistics about the numerical features\n", "df.describe().T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " \tmean | std\t| min\t| 25%\t| 50%\t| 75%\t| max\n", " 0. Unnamed: 0 mean | std\t| min\t| 25%\t| 50%\t| 75%\t| max\n", " 1. visits mean | std\t| min\t| 25%\t| 50%\t| 75%\t| max \n", " 2. nvisits mean | std\t| min\t| 25%\t| 50%\t| 75%\t| max \n", " 3. ovisits mean | std\t| min\t| 25%\t| 50%\t| 75%\t| max \n", " 4. novisits mean | std\t| min\t| 25%\t| 50%\t| 75%\t| max \n", " 5. emergency mean | std\t| min\t| 25%\t| 50%\t| 75%\t| max \n", " 6. hospital mean | std\t| min\t| 25%\t| 50%\t| 75%\t| max \n", " 7. chronic mean | std\t| min\t| 25%\t| 50%\t| 75%\t| max \n", " 8. age ???? | std\t| min\t| 25%\t| 50%\t| 75%\t| max \n", " 9. school int64 \n", " 10. income ???? | std\t| min\t| 25%\t| 50%\t| 75%\t| max " ] }, { "cell_type": "code", "execution_count": 89, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Unnamed: 0 int64\n", "visits int64\n", "nvisits int64\n", "ovisits int64\n", "novisits int64\n", "emergency int64\n", "hospital int64\n", "health object\n", "chronic int64\n", "adl object\n", "region object\n", "age float64\n", "gender object\n", "married object\n", "school int64\n", "income float64\n", "employed object\n", "insurance object\n", "medicaid object\n", "dtype: object" ] }, "execution_count": 89, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Display data types of each feature\n", "df.dtypes" ] }, { "cell_type": "code", "execution_count": 90, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Unnamed: 0 4406\n", "visits 60\n", "nvisits 51\n", "ovisits 37\n", "novisits 35\n", "emergency 11\n", "hospital 9\n", "health 3\n", "chronic 9\n", "adl 2\n", "region 4\n", "age 36\n", "gender 2\n", "married 2\n", "school 19\n", "income 3015\n", "employed 2\n", "insurance 2\n", "medicaid 2\n", "dtype: int64" ] }, "execution_count": 90, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Count of unique values in each column\n", "df.nunique()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## this is my reccomendation make adl, gender, married, employed, insurance, medicaid --> Booleans" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Export the data to JSON as NSMES1988.json" ] }, { "cell_type": "code", "execution_count": 91, "metadata": {}, "outputs": [], "source": [ "# save the DataFrame to a JSON file\n", "df.to_json('NSMES1988-NSMES1988.json', orient='records', lines=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Perform memory information on the data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index 132\n", "Unnamed: 0 35248\n", "visits 35248\n", "nvisits 35248\n", "ovisits 35248\n", "novisits 35248\n", "emergency 35248\n", "hospital 35248\n", "health 245760\n", "chronic 35248\n", "adl 243229\n", "region 242788\n", "age 35248\n", "gender 238774\n", "married 227112\n", "school 35248\n", "income 35248\n", "employed 225161\n", "insurance 228127\n", "medicaid 225108\n", "dtype: int64" ] }, "execution_count": 92, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Display memory usage of each feature in the DataFrame\n", "df.memory_usage(deep=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Recommend what non-default data types would you recommend to optimize memory settings for the dataframe" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# What changes would you recommend on the dataframe before attempting a detailed data analysis" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Export the data frame as a new CSV file NSMES1988new.csv" ] }, { "cell_type": "code", "execution_count": 93, "metadata": {}, "outputs": [], "source": [ "df.to_csv('NSMES1988new.csv', index=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Short Report\n", "### Pysical Inspection of the Data\n", " - \n", "### No Missing Values\n", "### Data types \n", "#### Values:\n", "#### Ranges:\n", "\n", "### I then exported as a JSON to \"NSMES1988-NSMES1988.json\"\n", "## Changes to the data I reccommend: \n", "- \n", "- " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.2" } }, "nbformat": 4, "nbformat_minor": 2 }