Skip to content

Use curl instead of wget for Spark and Julia downloads #1950

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Aug 3, 2023
Merged

Use curl instead of wget for Spark and Julia downloads #1950

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
34f5a18
Force tls version on some wget downloads
mathbunnyru Jul 26, 2023
7746493
Use TLSv1.3 for apache
mathbunnyru Jul 26, 2023
94712db
Try to use --retry-connrefused
mathbunnyru Jul 26, 2023
cd34331
Use run-one-until-success
mathbunnyru Jul 27, 2023
7612c90
Merge branch 'main' into asalikhov/force_tls
mathbunnyru Jul 28, 2023
a8e4880
Merge branch 'main' into asalikhov/force_tls
mathbunnyru Jul 28, 2023
b458b28
Merge branch 'main' into asalikhov/force_tls
mathbunnyru Jul 28, 2023
c265df3
Merge branch 'main' into asalikhov/force_tls
mathbunnyru Jul 28, 2023
d1dda4c
Merge branch 'main' into asalikhov/force_tls
mathbunnyru Jul 28, 2023
2da005b
Merge branch 'main' into asalikhov/force_tls
mathbunnyru Jul 28, 2023
6a3478d
Merge branch 'main' into asalikhov/force_tls
mathbunnyru Jul 28, 2023
5af7b8f
Merge branch 'main' into asalikhov/force_tls
mathbunnyru Jul 29, 2023
ce931b3
Merge branch 'main' into asalikhov/force_tls
mathbunnyru Jul 29, 2023
f3dd029
Merge branch 'main' into asalikhov/force_tls
mathbunnyru Jul 30, 2023
1390eda
Merge branch 'main' into asalikhov/force_tls
mathbunnyru Aug 2, 2023
75b76eb
Use curl for julia download
mathbunnyru Aug 2, 2023
a513d91
Use curl for spark download as well
mathbunnyru Aug 2, 2023
89fa3f3
Merge branch 'main' into asalikhov/force_tls
mathbunnyru Aug 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/using/selecting.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ It contains:

- Everything in `jupyter/base-notebook`
- Common useful utilities like
[curl](https://curl.se),
[git](https://git-scm.com/),
[nano](https://www.nano-editor.org/) (actually `nano-tiny`),
[tzdata](https://www.iana.org/time-zones),
Expand Down
1 change: 1 addition & 0 deletions minimal-notebook/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ USER root
RUN apt-get update --yes && \
apt-get install --yes --no-install-recommends \
# Common useful utilities
curl \
git \
nano-tiny \
tzdata \
Expand Down
3 changes: 2 additions & 1 deletion minimal-notebook/setup-scripts/setup-julia.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ JULIA_MAJOR_MINOR=$(echo "${JULIA_VERSION}" | cut -d. -f 1,2)
# Download and install Julia
cd /tmp
mkdir "/opt/julia-${JULIA_VERSION}"
wget --progress=dot:giga "https://julialang-s3.julialang.org/bin/linux/${JULIA_SHORT_ARCH}/${JULIA_MAJOR_MINOR}/${JULIA_INSTALLER}"
curl --progress-bar --location --output "${JULIA_INSTALLER}" \
"https://julialang-s3.julialang.org/bin/linux/${JULIA_SHORT_ARCH}/${JULIA_MAJOR_MINOR}/${JULIA_INSTALLER}"
tar xzf "${JULIA_INSTALLER}" -C "/opt/julia-${JULIA_VERSION}" --strip-components=1
rm "${JULIA_INSTALLER}"

Expand Down
6 changes: 4 additions & 2 deletions pyspark-notebook/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,11 @@ WORKDIR /tmp
# You need to use https://archive.apache.org/dist/ website if you want to download old Spark versions
# But it seems to be slower, that's why we use recommended site for download
RUN if [ -z "${scala_version}" ]; then \
wget --progress=dot:giga -O "spark.tgz" "https://dlcdn.apache.org/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz"; \
curl --progress-bar --location --output "spark.tgz" \
"https://dlcdn.apache.org/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz"; \
else \
wget --progress=dot:giga -O "spark.tgz" "https://dlcdn.apache.org/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz"; \
curl --progress-bar --location --output "spark.tgz" \
"https://dlcdn.apache.org/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz"; \
fi && \
echo "${spark_checksum} *spark.tgz" | sha512sum -c - && \
tar xzf "spark.tgz" -C /usr/local --owner root --group root --no-same-owner && \
Expand Down