Project Overview

This project consists of two simple Node.js scripts designed to process collections of job-related JSON files:

• Merges all JSON files in a specified folder into a single merged.json file.
• Reads the merged JSON data, removes duplicates, optionally filters jobs by different criteria (e.g., budget, applicants, client verification), and produces two outputs:
• merged.json-summary.json: A concise summary of each job.
• Aggregated skill statistics printed to the excel file.

The goal is to streamline the process of combining raw job feeds and generating useful summary reports and statistics for further analysis.

---

Prerequisites

• Node.js (v14 or later)
• npm (comes bundled with Node.js)

---

Installation

1. Clone or download this repository:

2. Install dependencies:

   npm install

---

Configuration

• Input folder for JSON files

  • In mergeJson.js, set the inputFolder constant to the directory containing your individual JSON files.

• Filtering options

  • In jobfeed.js, several filtering steps are commented out by default. You can enable or adjust these sections to refine which jobs are processed:
    • Remove jobs of a specific type (e.g., FIXED).
    • Filter out low-paying jobs under a certain hourly rate.
    • Include only jobs from payment-verified clients.
    • Exclude jobs with too many applicants.
    • Focus on specific keywords in title/description.

---

Usage

1. Merge JSON files

Run the merge script to combine all .json files from the configured folder into merged.json:

node mergeJson.js

Output: merged.json containing the concatenated array of job objects.

2. Generate summaries and statistics

With your new merged.json in place, process the job feed:

node jobfeed.js

Outputs:

• merged.json-summary.json – a trimmed-down version of each job with key fields (title, skills, client reviews/spent, hourly rates, applicants, duration, etc.).
• Console output showing:
  • Number of jobs read and de-duplicated.
  • Earliest and latest job dates and total days span.
  • A sorted list of skill-based statistics (count per day, average rates, applicant counts, duration, and verification ratios).

---

File Structure

├── mergeJson.js           # Script to merge JSON files
├── jobfeed.js             # Script to summarize and analyze merged data
├── merged.json            # Auto-generated merged data file
├── merged.json-summary.json # Auto-generated summary file
├── statistic.csv          # (Optional) CSV with additional statistics
├── package.json           # Project metadata and dependencies
└── package-lock.json      # Locked dependency tree

---

Extending the Project

• Additional filters: Uncomment and customize the filtering logic in jobfeed.js to match specific criteria (e.g., budget ranges, skill keywords, client history).

• Export options: Modify the script to output aggregated data in other formats (CSV, Excel) using libraries like json2csv or xlsx.

• Visualization: Integrate a charting library (e.g., Chart.js) to generate visual reports of skill demand and job trends.

---

Tips for Source JSON file for Statistics

This project doesn't support the collecting source JSON file. So You must collect it on Upwork.com manually.

Here's a screenshot how to collect JSON file on Upwork.com

Copy JSON data and paste it to the file in #json folder.

---

Happy coding!