CodeBook.Rmd

---
title: "CodeBook"
author: "David C Bruce"
date: "Thursday, March 12, 2015"
output: html_document
---
### Description of this document "CodeBook"

This Code Book is describes the data source for the *[Getting and Cleaning Data Course Project](https://class.coursera.org/getdata-012/human_grading/view/courses/973499/assessments/3/submissions)*, and the subsequent cleaned data.

##### List and Tables in this document

  * Directory structure of the data set is described in __*List 1*__ below.
  * Technical information about the files are described in __*Table 1*__ below.
  * Original / Descriptive feature names are listed in __*Table 2*__ below.
  * Some anomolous names are mentioned in the "Comments" column of __*Table 2*__.
  
Note these tables are repeated in the README file for the readers convenience.

### Background on assignment

The assignment is to create one R script called __run_analysis.R__ that does the following:

  1. Merges the training and the test sets to create one data set.
  2. Extracts only the measurements on the mean and standard deviation for each measurement. 
  3. Uses descriptive activity names to name the activities in the data set
  4. Appropriately labels the data set with descriptive variable names. 
  5. From the data set in step 4, creates a second, independent tidy data set with the average of each variable for each activity and each subject.
  
_Instructions for reading tidy data set from exercise grading page_

  1. Click on "wide_tidy_data" link in grading window
  2. New tab opens in browser with a bunch of unreadable text (write.table format)
  3. "Save as..." __data.txt__ in your working RStudio directory
  4. Run the following code in RStudio:

```{r eval=FALSE}
import_data<-read.table("data.txt", header=T)
View(import_data)
```

***
### Description of the study

The original data set is found on the UCI Machine Learning Repository under the study *[Human Activity Recognition Using Smartphones Data Set ](http://archive.ics.uci.edu/ml/datasets/Human+Activity+Recognition+Using+Smartphones)* (UCI HAR)

Directly quoting information found on the UCI HAR website "The experiments have been carried out with a group of 30 volunteers within an age bracket of 19-48 years. Each person performed six activities (WALKING, WALKING_UPSTAIRS, WALKING_DOWNSTAIRS, SITTING, STANDING, LAYING) wearing a smartphone (Samsung Galaxy S II) on the waist. Using its embedded accelerometer and gyroscope, we captured 3-axial linear acceleration and 3-axial angular velocity at a constant rate of 50Hz. The experiments have been video-recorded to label the data manually. The obtained data set has been randomly partitioned into two sets, where 70% of the volunteers was selected for generating the training data and 30% the test data."

***
### Discussion of the Original Data set

The variables in the UCI data set are activity, subject, and data generated by an embedded accelerometer (3-axial linear acceleration) and gyroscope (3-axial angular velocity).

  * Data was collected over time, so the data is represented in the time domain ("In Time")
  * All features are descriptive summaries of the accelerometer and gyroscope data.
  * The accelerometer data was decomposed into body and gravity values through a filtering procedure
  * Jerk signals were derived from time data: body linear acceleration (first derivative of acceleration) and angular velocity (second derivative of velocity).  Presumably the activities have characteristic jerk values that can be used for machine learning
  * The accelerometer and gyroscope data sets were transformed using a fast Fourier transform (FFT).  This transoforms the data to the frequency domain ("In Freq") Presumably the activities have characteristic frequency profiles that can be used for machine learning.

Note: The data for the seven "angle" features were included in the tidy data set because the name "Mean" appears in the feature name.  Presumably the "angle" features and the "magnitude" features are vector elements of the corresponding features.

### Discussion of the Consolidated Data Set

The variables in the UCI data set are:

  * data_type (train or test)
  * activity (six values)
  * subject (30 values)
  * Accelerometer data (3-axial linear acceleration)
  * Gyroscope data (3-axial angular velocity)

Accelerometer and Gyroscope data have been summarized in a number of features.  The two summary types we are evaluating in this exercise are Mean signal (mean()), and Standard Deviation of the Mean (std()). The raw Accelerometer and Gyroscope data and the other features are ignored in this exercise.

Data sets were merged and renamed as described in detail in the README.

In brief:

  * Data sets were identified as "test",or "train" in a new column named "data_type."  (This column was not used in the final tidy data set)
  * Activity and subject data were added to the corresponding feature tables
  * Activity codes were replaced with activity names
  * Feature names were were renamed more descriptively.
    * Note in particular, the sensor type codes "Acc" and "Gyro" were replaced with the phenomona that was actually measured, "linear acceleration" for Acc, and "angular velocity" for Gyro.  Original / Descriptive feature names are listed in __*Table 2*__ below.
    * Some anomolous names are mentioned in the "Comments" column of Table 2.
  * Test and train datasets were merged using "rbind."

The final set is 10,299 observations of 89 variables.

### Discussion of the Tidy set

The Condolidated Data set was wrangled:

```{r eval=FALSE}
library(tidyr)
library(dplyr)
library(reshape)
...

consolidated_data_melt<-melt(consolidated_data, id=c(2:3), measure=c(4:89))
...
wide_tidy_data<-cast(consolidated_data_melt, activity+subject ~ variable, mean)
wide_tidy_data<-cast(consolidated_data_melt, activity+subject ~ variable, mean)
```


into two tidy data files:

  * "Wide" format file tidy data set with the average of each variable for each activity and each subject.
    * 180 Observations by 88 variables
    * __Data set submitted for evaluation__
    * Variables described in in __*List 2*__ below.

  * "Long" format file tidy data set with the average of each variable for each activity and each subject.
    * 15,840 observations by 4 variables
    * __Data set not used for assignment__, but included in the repo.

***
#### List 1: Directory structure of original data
* **UCI HAR Dataset**
    * activity_labels.txt
    * features.txt
    * features_info.txt
    * README.txt
    * **test**
        * subject_test.txt
        * X_test.txt
        * y_test.txt
        * **Inertial Signals**
            * (Raw data files not used in this exercise)
    * **train**
        * subject_train.txt
        * X_train.txt
        * y_train.txt 
        * **Inertial Signals**
            * (Raw data files not used in this exercise)

***
#### Table 1: Contents of original data files

Directory | File | Rows | Columns | Description
--------- | ---- | ---- | ------- | -----------
UCI HAR Dataset | activity_labels.txt | 6 | 2 | First column is key field (integer).  The  second column is an activity, such as "WALKING" (string)
UCI HAR Dataset | features.txt | 561 | 2 | First column is key field (integer).  The second column is a sensor parameter-summary descriptive statistic-dimension, such as "tBodyAcc-mean()-X" (numeric)
UCI HAR Dataset/test | subject_test.txt | 2947 | 1 |Test Subject identifier for each observation (integer)
UCI HAR Dataset/test | X_test.txt | 2947 | 561 | Test observation features (numeric)  Note: column labels are found in features.txt and X_test column numbers correspond to feature key rows.
UCI HAR Dataset/test | y_test.txt | 2947 | 1 | Test Activity for each observation (Integer and row keyed to activity_label)
UCI HAR Dataset/train | subject_train.txt | 7352 | 1 | Train Subject identifier for each observation (integer)
UCI HAR Dataset/train | X_train.txt | 7352 | 561 | Train observation features (numeric)  Note: column labels are found in features.txt and X_test column numbers correspond to feature key rows.
UCI HAR Dataset/train | y_train.txt | 7352 | 1 |  Train Activity for each observation (Integer and row keyed to activity_label)

***
#### Table 2: Feature codes and Feature descriptive renames in consolidated data table

Original Data Name | Descriptive Data Name | Comments
------------------ | --------------------- | --------
data_type | data_type | 
subject | subject | 
activity | activity | 
tBodyAcc-mean()-X | Body Linear_Acceleration in_X  in_Time  Mean | 
tBodyAcc-mean()-Y | Body Linear_Acceleration in_Y  in_Time  Mean | 
tBodyAcc-mean()-Z | Body Linear_Acceleration in_Z  in_Time  Mean | 
tGravityAcc-mean()-X | Gravity Linear_Acceleration in_X  in_Time  Mean | 
tGravityAcc-mean()-Y | Gravity Linear_Acceleration in_Y  in_Time  Mean | 
tGravityAcc-mean()-Z | Gravity Linear_Acceleration in_Z  in_Time  Mean | 
tBodyAccJerk-mean()-X | Body Linear_Acceleration Jerk in_X  in_Time  Mean | 
tBodyAccJerk-mean()-Y | Body Linear_Acceleration Jerk in_Y  in_Time  Mean | 
tBodyAccJerk-mean()-Z | Body Linear_Acceleration Jerk in_Z  in_Time  Mean | 
tBodyGyro-mean()-X | Body Angular_Velocity in_X  in_Time  Mean | 
tBodyGyro-mean()-Y | Body Angular_Velocity in_Y  in_Time  Mean | 
tBodyGyro-mean()-Z | Body Angular_Velocity in_Z  in_Time  Mean | 
tBodyGyroJerk-mean()-X | Body Angular_Velocity Jerk in_X  in_Time  Mean | 
tBodyGyroJerk-mean()-Y | Body Angular_Velocity Jerk in_Y  in_Time  Mean | 
tBodyGyroJerk-mean()-Z | Body Angular_Velocity Jerk in_Z  in_Time  Mean | 
tBodyAccMag-mean() | Body Linear_Acceleration Magnitude  in_Time  Mean | 
tGravityAccMag-mean() | Gravity Linear_Acceleration Magnitude  in_Time  Mean | 
tBodyAccJerkMag-mean() | Body Linear_Acceleration Jerk Magnitude  in_Time  Mean | 
tBodyGyroMag-mean() | Body Angular_Velocity Magnitude  in_Time  Mean | 
tBodyGyroJerkMag-mean() | Body Angular_Velocity Jerk Magnitude  in_Time  Mean | 
fBodyAcc-mean()-X | Body Linear_Acceleration in_X  in_Freq  Mean | 
fBodyAcc-mean()-Y | Body Linear_Acceleration in_Y  in_Freq  Mean | 
fBodyAcc-mean()-Z | Body Linear_Acceleration in_Z  in_Freq  Mean | 
fBodyAccJerk-mean()-X | Body Linear_Acceleration Jerk in_X  in_Freq  Mean | 
fBodyAccJerk-mean()-Y | Body Linear_Acceleration Jerk in_Y  in_Freq  Mean | 
fBodyAccJerk-mean()-Z | Body Linear_Acceleration Jerk in_Z  in_Freq  Mean | 
fBodyGyro-mean()-X | Body Angular_Velocity in_X  in_Freq  Mean | 
fBodyGyro-mean()-Y | Body Angular_Velocity in_Y  in_Freq  Mean | 
fBodyGyro-mean()-Z | Body Angular_Velocity in_Z  in_Freq  Mean | 
fBodyAccMag-mean() | Body Linear_Acceleration Magnitude  in_Freq  Mean | 
fBodyBodyAccJerkMag-mean() | Body BodyLinear_Acceleration Jerk Magnitude  in_Freq  Mean | 
fBodyBodyGyroMag-mean() | Body BodyAngular_Velocity Magnitude  in_Freq  Mean | 
fBodyBodyGyroJerkMag-mean() | Body BodyAngular_Velocity Jerk Magnitude  in_Freq  Mean | 
tBodyAcc-std()-X | Body Linear_Acceleration in_X  in_Time  StDev | 
tBodyAcc-std()-Y | Body Linear_Acceleration in_Y  in_Time  StDev | 
tBodyAcc-std()-Z | Body Linear_Acceleration in_Z  in_Time  StDev | 
tGravityAcc-std()-X | Gravity Linear_Acceleration in_X  in_Time  StDev | 
tGravityAcc-std()-Y | Gravity Linear_Acceleration in_Y  in_Time  StDev | 
tGravityAcc-std()-Z | Gravity Linear_Acceleration in_Z  in_Time  StDev | 
tBodyAccJerk-std()-X | Body Linear_Acceleration Jerk in_X  in_Time  StDev | 
tBodyAccJerk-std()-Y | Body Linear_Acceleration Jerk in_Y  in_Time  StDev | 
tBodyAccJerk-std()-Z | Body Linear_Acceleration Jerk in_Z  in_Time  StDev | 
tBodyGyro-std()-X | Body Angular_Velocity in_X  in_Time  StDev | 
tBodyGyro-std()-Y | Body Angular_Velocity in_Y  in_Time  StDev | 
tBodyGyro-std()-Z | Body Angular_Velocity in_Z  in_Time  StDev | 
tBodyGyroJerk-std()-X | Body Angular_Velocity Jerk in_X  in_Time  StDev | 
tBodyGyroJerk-std()-Y | Body Angular_Velocity Jerk in_Y  in_Time  StDev | 
tBodyGyroJerk-std()-Z | Body Angular_Velocity Jerk in_Z  in_Time  StDev | 
tBodyAccMag-std() | Body Linear_Acceleration Magnitude  in_Time  StDev | 
tGravityAccMag-std() | Gravity Linear_Acceleration Magnitude  in_Time  StDev | 
tBodyAccJerkMag-std() | Body Linear_Acceleration Jerk Magnitude  in_Time  StDev | 
tBodyGyroMag-std() | Body Angular_Velocity Magnitude  in_Time  StDev | 
tBodyGyroJerkMag-std() | Body Angular_Velocity Jerk Magnitude  in_Time  StDev | 
fBodyAcc-std()-X | Body Linear_Acceleration in_X  in_Freq  StDev | 
fBodyAcc-std()-Y | Body Linear_Acceleration in_Y  in_Freq  StDev | 
fBodyAcc-std()-Z | Body Linear_Acceleration in_Z  in_Freq  StDev | 
fBodyAccJerk-std()-X | Body Linear_Acceleration Jerk in_X  in_Freq  StDev | 
fBodyAccJerk-std()-Y | Body Linear_Acceleration Jerk in_Y  in_Freq  StDev | 
fBodyAccJerk-std()-Z | Body Linear_Acceleration Jerk in_Z  in_Freq  StDev | 
fBodyGyro-std()-X | Body Angular_Velocity in_X  in_Freq  StDev | 
fBodyGyro-std()-Y | Body Angular_Velocity in_Y  in_Freq  StDev | 
fBodyGyro-std()-Z | Body Angular_Velocity in_Z  in_Freq  StDev | 
fBodyAccMag-std() | Body Linear_Acceleration Magnitude  in_Freq  StDev | 
fBodyBodyAccJerkMag-std() | Body BodyLinear_Acceleration Jerk Magnitude  in_Freq  StDev | 
fBodyBodyGyroMag-std() | Body BodyAngular_Velocity Magnitude  in_Freq  StDev | 
fBodyBodyGyroJerkMag-std() | Body BodyAngular_Velocity Jerk Magnitude  in_Freq  StDev | 
fBodyAcc-meanFreq()-X | Body Linear_Acceleration in_X  in_Freq  Mean_Freq | 
fBodyAcc-meanFreq()-Y | Body Linear_Acceleration in_Y  in_Freq  Mean_Freq | 
fBodyAcc-meanFreq()-Z | Body Linear_Acceleration in_Z  in_Freq  Mean_Freq | 
fBodyAccJerk-meanFreq()-X | Body Linear_Acceleration Jerk in_X  in_Freq  Mean_Freq | 
fBodyAccJerk-meanFreq()-Y | Body Linear_Acceleration Jerk in_Y  in_Freq  Mean_Freq | 
fBodyAccJerk-meanFreq()-Z | Body Linear_Acceleration Jerk in_Z  in_Freq  Mean_Freq | 
fBodyGyro-meanFreq()-X | Body Angular_Velocity in_X  in_Freq  Mean_Freq | 
fBodyGyro-meanFreq()-Y | Body Angular_Velocity in_Y  in_Freq  Mean_Freq | 
fBodyGyro-meanFreq()-Z | Body Angular_Velocity in_Z  in_Freq  Mean_Freq | 
fBodyAccMag-meanFreq() | Body Linear_Acceleration Magnitude  in_Freq  Mean_Freq | 
fBodyBodyAccJerkMag-meanFreq() | Body BodyLinear_Acceleration Jerk Magnitude  in_Freq  Mean_Freq | Not clear on what Body Body refers to.  Possible naming error?
fBodyBodyGyroMag-meanFreq() | Body BodyAngular_Velocity Magnitude  in_Freq  Mean_Freq |  Not clear on what Body Body refers to.  Possible naming error?
fBodyBodyGyroJerkMag-meanFreq() | Body BodyAngular_Velocity Jerk Magnitude  in_Freq  Mean_Freq |  Not clear on what Body Body refers to.  Possible naming error?
angle(tBodyAccMean,gravity) | angle(tBody Linear_Acceleration Mean,gravity) | Partially modified angle name. Note possible naming error. "gravity" may be "gravityMean" as in other angle features
angle(tBodyAccJerkMean),gravityMean) | angle(tBody Linear_Acceleration Jerk Mean),gravityMean) |  Partially modified angle name.Note two right parenthesis - naming error?
angle(tBodyGyroMean,gravityMean) | angle(tBody Angular_Velocity Mean,gravityMean) |  Partially modified angle name. 
angle(tBodyGyroJerkMean,gravityMean) | angle(tBody Angular_Velocity Jerk Mean,gravityMean) |  Partially modified angle name. 
angle(X,gravityMean) | x_angle(X,gravityMean) | x_ added as hack to force angle features to sort to the end of any list
angle(Y,gravityMean) | x_angle(Y,gravityMean) | x_ added as hack to force angle features to sort to the end of any list
angle(Z,gravityMean) | x_angle(Z,gravityMean) | x_ added as hack to force angle features to sort to the end of any list