Skip to content

Commit 069643d

Browse files
zachrattnerzachrattner
committed
Initial commit
0 parents  commit 069643d

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

62 files changed

+15184
-0
lines changed

‎.gitignore

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Node modules
2+
node_modules/
3+
4+
# Build output
5+
dist/
6+
out/
7+
build/
8+
release/
9+
10+
# Logs
11+
npm-debug.log*
12+
yarn-debug.log*
13+
yarn-error.log*
14+
pnpm-debug.log*
15+
16+
# Environment variables
17+
.env
18+
.env.local
19+
.env.*.local
20+
21+
# OS-specific files
22+
.DS_Store
23+
Thumbs.db
24+
25+
# TypeScript
26+
*.tsbuildinfo
27+
28+
# IDE-specific files
29+
.vscode/
30+
.idea/
31+
*.swp
32+
33+
# Temporary files
34+
*.tmp
35+
*.temp
36+
37+
# Electron-specific files
38+
*.asar
39+
40+
# Scour files
41+
*.scour
42+
43+
# Playwright cache
44+
browsers
45+
46+
# Helper scripts
47+
setup-git.sh

‎LICENSE

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Copyright (c) 2024-present Worthwhile Adventures LLC
2+
3+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
4+
5+
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
6+
7+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

‎README.md

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Scourhead
2+
3+
Scourhead is an open-source agentic AI application designed to streamline online research by automating tedious workflows. It leverages the Llama 3.2 LLM and Playwright for browser automation to deliver structured search results into a spreadsheet, mimicking how a skilled human would perform the research task.
4+
5+
[![AI Labels - Made by Humans with AI](https://ailabels.org/badges/AI%20Labels%20-%20Human%20and%20AI.svg "Made By Humans with AI")](https://ailabels.org)
6+
7+
## Features
8+
9+
- **Agentic AI Automation**: Define your research objective, and Scourhead handles the rest, from query generation to data extraction. Two agents review the search results, one for checking if the page is relevant, and a second to extract the requested fields in structured form.
10+
- **Customizable Outputs**: Specify the data fields and format that meet your needs. The LLM parses the page and converts into the format you request.
11+
- **Browser Automation**: Uses Playwright to navigate search results and extract data from relevant pages.
12+
- **Local Execution**: Runs entirely on your machine with no backend server, ensuring your data stays private.
13+
- **Lightweight**: Optimized to run efficiently on consumer-grade devices using Meta's Llama 3.2 (3B) model.
14+
15+
## File Structure
16+
17+
```
18+
scourhead/
19+
├── icons/ # Icons for the app
20+
├── src/ # Source code
21+
├── package.json # Project metadata and dependencies
22+
├── package-lock.json # Dependency tree lockfile
23+
├── README.md # Project documentation (this file)
24+
├── tsconfig.json # TypeScript configuration file
25+
```
26+
27+
## Getting Started
28+
29+
### Prerequisites
30+
- **Node.js**: Ensure you have Node.js installed on your machine.
31+
- **Playwright**: The project uses Playwright for browser automation.
32+
- **Ollama**: Required for running large language models locally.
33+
34+
### Installation
35+
1. Clone the repository:
36+
```bash
37+
git clone https://github.com/zachrattner/scourhead.git
38+
cd scourhead
39+
```
40+
2. Install dependencies:
41+
```bash
42+
npm install
43+
```
44+
3. Install headless Chromium browser from Playwright:
45+
```bash
46+
./download-chromium.sh
47+
```
48+
4. Ensure Ollama is installed and running.
49+
50+
## Usage
51+
1. Define your objective. Tell Scourhead the context and goals just like you would a human assistant.
52+
2. Review Scourhead's search query suggestions, and make edits if needed
53+
3. Perform the search and review results in realtime as they come back
54+
4. Process each page via LLM and review results in realtime
55+
5. Export the results as CSV
56+
57+
If you would like to review a completed project, open [sample.scour](examples/sample.scour) in Scourhead or review [sample.csv](examples/sample.csv) to see what an example export looks like.
58+
59+
## License
60+
This project is licensed under the MIT License - see the `LICENSE` file for details.
61+
62+
## Contributing
63+
Contributions are welcome! Please open an issue or submit a pull request to propose improvements or report bugs.
64+
65+
## Disclaimers
66+
1. I have never written an Electron app before. I might not have done things in an idiomatic way.
67+
2. I limited myself to develop only during the week of Christmas between family obligations. There is a lot that could be improved and I just haven't had time yet.
68+
3. The output format might not 100% match what you requested. LLMs are not perfect. Just think of all the time you are not spending manually searching when you are making corrections 😎
69+
4. I've been told some search engines don't like being used this way. You're responsible for what you do with this app.
70+
71+
## Contact
72+
Feel free to [join Scourhead on Discord](https://discord.gg/N9NKfSDWme) or [hit me up on LinkedIn](https://linkedin.com/in/zachrattner).
73+
74+
## Copyright
75+
(c) 2024-present Worthwhile Adventures LLC

‎do-release.sh

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
#!/bin/zsh
2+
3+
# ------------------------------------------------------------------------------
4+
# DO RELEASE
5+
# ------------------------------------------------------------------------------
6+
# This script automates the release process:
7+
# 1. Builds the application for macOS, Windows, and Linux
8+
# 2. Uploads the build artifacts to AWS S3
9+
# 3. Invalidates the CloudFront cache for the release files
10+
# 4. Tags the latest commit on main with the version number
11+
# 5. Creates a GitHub release for the version number
12+
# 6. Updates the download links in the index.html file
13+
# 7. Invalidates the CloudFront cache for the index.html file
14+
# ------------------------------------------------------------------------------
15+
# Prerequisites:
16+
# 1. jq: Command-line JSON processor
17+
# 2. AWS CLI: For interacting with AWS S3
18+
# 3. GitHub CLI (gh): For tagging commits and creating releases on GitHub
19+
#
20+
# Install these tools on macOS using Homebrew:
21+
# - jq: brew install jq
22+
# - AWS CLI: brew install awscli
23+
# - GitHub CLI: brew install gh
24+
#
25+
# Additional Setup:
26+
# - Configure AWS CLI with `aws configure` and set up a profile named
27+
# 'worthwhileadventures'.
28+
# - Authenticate GitHub CLI with `gh auth login`.
29+
# ------------------------------------------------------------------------------
30+
31+
# Enable strict mode
32+
set -euo pipefail
33+
34+
# Extract version from package.json
35+
APP_VERSION=$(jq -r '.version' package.json)
36+
echo "Detected application version: ${APP_VERSION}"
37+
38+
# S3 bucket and profile
39+
S3_BUCKET="s3://scourhead.com"
40+
S3_RELEASE_FOLDER="$S3_BUCKET/releases"
41+
AWS_PROFILE="worthwhileadventures"
42+
BASE_URL="https://scourhead.com/releases"
43+
CLOUDFRONT_DISTRIBUTION_ID="E146D77OPFI37E"
44+
45+
# Function to print errors and exit
46+
error_exit() {
47+
echo "Error: $1" >&2
48+
exit 1
49+
}
50+
51+
# Check if a Git tag exists
52+
check_and_create_tag() {
53+
local tag="v${APP_VERSION}"
54+
if ! git rev-parse "$tag" > /dev/null 2>&1; then
55+
echo "Creating Git tag: $tag"
56+
git tag -a "$tag" -m "Release ${APP_VERSION}" || error_exit "Failed to create tag $tag"
57+
git push origin "$tag" || error_exit "Failed to push tag $tag"
58+
else
59+
echo "Git tag $tag already exists. Skipping."
60+
fi
61+
}
62+
63+
# Create a GitHub release
64+
create_github_release() {
65+
local tag="v${APP_VERSION}"
66+
local release_notes=$(cat <<EOF
67+
# Scourhead v${APP_VERSION}
68+
69+
Download links for all architectures:
70+
71+
- [macOS arm64](https://scourhead.com/releases/${APP_VERSION}/Scourhead-${APP_VERSION}-arm64.dmg)
72+
- [Windows x64](https://scourhead.com/releases/${APP_VERSION}/Scourhead-Setup-x64.exe)
73+
- [Windows arm64](https://scourhead.com/releases/${APP_VERSION}/Scourhead-Setup-arm64.exe)
74+
- [Linux x64](https://scourhead.com/releases/${APP_VERSION}/scourhead_${APP_VERSION}_amd64.deb)
75+
- [Linux arm64](https://scourhead.com/releases/${APP_VERSION}/scourhead_${APP_VERSION}_arm64.deb)
76+
EOF
77+
)
78+
79+
if ! gh release view "$tag" > /dev/null 2>&1; then
80+
echo "Creating GitHub release for $tag"
81+
gh release create "$tag" \
82+
--title "v${APP_VERSION}" \
83+
--notes "$release_notes" || error_exit "Failed to create GitHub release"
84+
else
85+
echo "GitHub release $tag already exists. Skipping."
86+
fi
87+
}
88+
89+
cleanup() {
90+
echo "Cleaning up release directory..."
91+
rm -rf release && npm run clean || error_exit "Failed to clean up release directory"
92+
}
93+
94+
build_upload_check() {
95+
local build_command=$1
96+
local file_path=$2
97+
local s3_path=$3
98+
99+
echo "Running build command: $build_command"
100+
eval "$build_command" || error_exit "Build command failed: $build_command"
101+
102+
echo "Uploading $file_path to $s3_path"
103+
aws s3 cp "$file_path" "$s3_path" --profile "$AWS_PROFILE" || error_exit "Failed to upload $file_path to S3"
104+
105+
echo "Verifying upload: $s3_path"
106+
aws s3 ls "$s3_path" --profile "$AWS_PROFILE" > /dev/null || error_exit "Uploaded file not found in S3: $s3_path"
107+
}
108+
109+
invalidate_cloudfront_cache() {
110+
echo "Invalidating CloudFront cache for /releases/${APP_VERSION}/*"
111+
INVALIDATION_ID=$(aws cloudfront create-invalidation \
112+
--distribution-id "$CLOUDFRONT_DISTRIBUTION_ID" \
113+
--paths "/releases/${APP_VERSION}/*" \
114+
--profile "$AWS_PROFILE" \
115+
--query 'Invalidation.Id' \
116+
--output text) || error_exit "Failed to create CloudFront invalidation"
117+
118+
echo "CloudFront invalidation created: $INVALIDATION_ID"
119+
}
120+
121+
update_index_html() {
122+
echo "Downloading index.html from S3..."
123+
aws s3 cp "$S3_BUCKET/index.html" index.html --profile "$AWS_PROFILE" || error_exit "Failed to download index.html"
124+
125+
echo "Updating download links in index.html..."
126+
sed -i '' -e "s#https://scourhead.com/releases/.*/Scourhead-[0-9.]*-arm64.dmg#https://scourhead.com/releases/${APP_VERSION}/Scourhead-${APP_VERSION}-arm64.dmg#g" index.html
127+
sed -i '' -e "s#https://scourhead.com/releases/.*/Scourhead-Setup-x64.exe#https://scourhead.com/releases/${APP_VERSION}/Scourhead-Setup-x64.exe#g" index.html
128+
sed -i '' -e "s#https://scourhead.com/releases/.*/Scourhead-Setup-arm64.exe#https://scourhead.com/releases/${APP_VERSION}/Scourhead-Setup-arm64.exe#g" index.html
129+
sed -i '' -e "s#https://scourhead.com/releases/.*/scourhead_[0-9.]*_amd64.deb#https://scourhead.com/releases/${APP_VERSION}/scourhead_${APP_VERSION}_amd64.deb#g" index.html
130+
sed -i '' -e "s#https://scourhead.com/releases/.*/scourhead_[0-9.]*_arm64.deb#https://scourhead.com/releases/${APP_VERSION}/scourhead_${APP_VERSION}_arm64.deb#g" index.html
131+
132+
echo "Uploading updated index.html to S3..."
133+
aws s3 cp index.html "$S3_BUCKET/index.html" --profile "$AWS_PROFILE" || error_exit "Failed to upload updated index.html"
134+
135+
echo "Invalidating CloudFront cache for /index.html..."
136+
INVALIDATION_ID=$(aws cloudfront create-invalidation \
137+
--distribution-id "$CLOUDFRONT_DISTRIBUTION_ID" \
138+
--paths "/index.html" \
139+
--profile "$AWS_PROFILE" \
140+
--query 'Invalidation.Id' \
141+
--output text) || error_exit "Failed to create CloudFront invalidation for /index.html"
142+
143+
echo "CloudFront invalidation created for /index.html: $INVALIDATION_ID"
144+
145+
rm index.html
146+
}
147+
148+
cleanup
149+
build_upload_check "npm run build-mac" \
150+
"release/Scourhead-${APP_VERSION}-arm64.dmg" \
151+
"$S3_RELEASE_FOLDER/${APP_VERSION}/Scourhead-${APP_VERSION}-arm64.dmg"
152+
153+
cleanup
154+
build_upload_check "npm run build-win-x64" \
155+
"release/Scourhead Setup ${APP_VERSION}.exe" \
156+
"$S3_RELEASE_FOLDER/${APP_VERSION}/Scourhead-Setup-x64.exe"
157+
158+
cleanup
159+
build_upload_check "npm run build-win-arm64" \
160+
"release/Scourhead Setup ${APP_VERSION}.exe" \
161+
"$S3_RELEASE_FOLDER/${APP_VERSION}/Scourhead-Setup-arm64.exe"
162+
163+
cleanup
164+
build_upload_check "npm run build-linux-x64" \
165+
"release/scourhead_${APP_VERSION}_amd64.deb" \
166+
"$S3_RELEASE_FOLDER/${APP_VERSION}/scourhead_${APP_VERSION}_amd64.deb"
167+
168+
cleanup
169+
build_upload_check "npm run build-linux-arm64" \
170+
"release/scourhead_${APP_VERSION}_arm64.deb" \
171+
"$S3_RELEASE_FOLDER/${APP_VERSION}/scourhead_${APP_VERSION}_arm64.deb"
172+
173+
cleanup
174+
invalidate_cloudfront_cache
175+
check_and_create_tag
176+
create_github_release
177+
update_index_html
178+
179+
echo "All tasks completed successfully!"

0 commit comments

Comments
 (0)