Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Example Dockerfile fixes for PolyTracker short tool paper #6570

Merged
merged 9 commits into from
Jul 3, 2024
Merged

Example Dockerfile fixes for PolyTracker short tool paper #6570

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f27fadb
libjpeg: stable sources link
kaoudis Jun 28, 2024
b0896cb
nitro: custom build system of nitro breaks if old python isnt availab…
kaoudis Jun 28, 2024
7e02a88
png: add missing wget dependency
kaoudis Jun 28, 2024
8031618
xpdf: fix zlib location (stored old versions modev to /fossils on the…
kaoudis Jun 28, 2024
6f69fdd
qpdf: use ijg stable jpeg source url
kaoudis Jun 28, 2024
77aba58
file: fix example Dockerfile by checking out a tag from reasonably ar…
kaoudis Jul 2, 2024
314b469
xpdf: make everything work with modern polytracker, but a bit bigger …
kaoudis Jul 3, 2024
367eb74
xpdf: one big instrument-targets
kaoudis Jul 3, 2024
9f38008
xpdf: comments
kaoudis Jul 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions examples/Dockerfile-file.demo
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,18 @@ MAINTAINER Carson Harmon <[email protected]>
WORKDIR /polytracker/the_klondike

RUN apt update
RUN apt-get install automake libtool make python zlib1g-dev git -y
RUN git clone https://github.com/file/file.git
RUN apt-get install automake libtool make zlib1g-dev git -y

RUN echo "temp" > /PLACEHOLDER
ENV POLYPATH=/PLACEHOLDER

#=================================
WORKDIR file
RUN autoreconf -f -i
WORKDIR /polytracker/the_klondike
RUN git clone https://github.com/file/file.git
WORKDIR /polytracker/the_klondike/file
RUN git fetch --tags && \
git checkout tags/FILE5_41
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this just pins a file version that seems compatible with when this dockerfile was originally created. I dont think we pinned a version before.

RUN autoreconf -fiv
RUN ./configure --prefix=/polytracker/the_klondike/bin/ --disable-shared
RUN polytracker build make -j$((`nproc`+1)) install

WORKDIR /polytracker/the_klondike/bin/bin
RUN polytracker instrument-targets --taint --ftrace file --ignore-lists libz
RUN mv file.instrumented file_track
3 changes: 2 additions & 1 deletion examples/Dockerfile-libjpeg.demo
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@ FROM ubuntu:focal AS libjpeg-sources
WORKDIR /polytracker/the_klondike/
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get -y upgrade && apt-get install -y wget
RUN wget http://jpegclub.org/reference/wp-content/uploads/2022/01/jpegsrc.v9e.tar.gz && tar xf jpegsrc.v9e.tar.gz
RUN wget https://www.ijg.org/files/jpegsrc.v9e.tar.gz && \
tar xvf jpegsrc.v9e.tar.gz

# Now, build the libjpeg image using previously downloaded source
FROM trailofbits/polytracker:latest
Expand Down
2 changes: 1 addition & 1 deletion examples/Dockerfile-nitro-nitf.demo
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ WORKDIR /polytracker/the_klondike/nitro/build
RUN polytracker build cmake .. \
-DCMAKE_C_FLAGS="-w -D_POSIX_C_SOURCE=200809L -DCODA_OSS_NO_is_trivially_copyable" \
-DCMAKE_CXX_FLAGS="-w -D_POSIX_C_SOURCE=200809L -DCODA_OSS_NO_is_trivially_copyable" \
-DCODA_BUILD_TESTS=OFF
-DCODA_BUILD_TESTS=OFF -DENABLE_PYTHON=OFF

RUN polytracker build cmake --build . -j$((`nproc`+1)) --target show_nitf++

Expand Down
2 changes: 1 addition & 1 deletion examples/Dockerfile-png.demo
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ RUN sed -Ei 's/^# deb-src /deb-src /' /etc/apt/sources.list
#Update pkg-config/util-linux (needed for FontConfig)
RUN apt update
RUN apt install pkg-config uuid-dev gperf libtool \
gettext autopoint autoconf -y
gettext autopoint autoconf wget -y

RUN apt-get install python3-dev

Expand Down
2 changes: 1 addition & 1 deletion examples/Dockerfile-qpdf.demo
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ WORKDIR /polytracker/the_klondike
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y git wget
RUN git clone --depth=1 --branch 11.5 https://github.com/qpdf/qpdf.git
RUN wget http://jpegclub.org/reference/wp-content/uploads/2022/01/jpegsrc.v9e.tar.gz && tar xf jpegsrc.v9e.tar.gz
RUN wget https://www.ijg.org/files/jpegsrc.v9e.tar.gz && tar xf jpegsrc.v9e.tar.gz

# Now, build the qpdf image using previously downloaded source
FROM trailofbits/polytracker:latest
Expand Down
104 changes: 48 additions & 56 deletions examples/Dockerfile-xpdf.demo
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
FROM trailofbits/polytracker
MAINTAINER Carson Harmon <[email protected]>
LABEL org.opencontainers.image.authors="[email protected],[email protected]"

WORKDIR /polytracker/the_klondike

#Update pkg-config/util-linux (needed for FontConfig)
RUN apt update && apt install pkg-config \
# Updating pkg-config/util-linux is needed for FontConfig
# Note also that qt5 is needed for the cmake build of xpdf
# (there appears to not be a cmake macro to turn off the build parts that require it)
RUN apt-get update && apt-get install -y \
pkg-config \
uuid-dev \
gperf \
wget \
Expand All @@ -18,89 +21,78 @@ RUN apt update && apt install pkg-config \
cmake \
libfreetype6-dev \
libxcb-composite0-dev \
libxml2-dev -y
libxml2-dev \
qtbase5-dev

#RUN apt-get install python3-dev

#Fontconfig requires some stuff?
#RUN apt install pkg-config \libasound2-dev libssl-dev cmake libfreetype6-dev libexpat1-dev libxcb-composite0-dev -y
#RUN apt install libxml2-dev -y
WORKDIR /polytracker/the_klondike
RUN wget https://dl.xpdfreader.com/old/xpdf-4.03.tar.gz
RUN tar -xvf xpdf-4.03.tar.gz
ENV build_dir=/polytracker/the_klondike/xpdf-4.03

#=================================
WORKDIR /polytracker/the_klondike
WORKDIR $build_dir

#FreeType http://www.linuxfromscratch.org/blfs/view/svn/general/freetype2.html
# FreeType http://www.linuxfromscratch.org/blfs/view/svn/general/freetype2.html
RUN wget https://downloads.sourceforge.net/freetype/freetype-2.10.1.tar.xz
RUN tar -xvf freetype-2.10.1.tar.xz

WORKDIR freetype-2.10.1
WORKDIR $build_dir/freetype-2.10.1

#Some linux from scratch magic
# Some linux from scratch magic
RUN sed -ri "s:.*(AUX_MODULES.*valid):\1:" modules.cfg
RUN sed -r "s:.*(#.*SUBPIXEL_RENDERING) .*:\1:" -i include/freetype/config/ftoption.h
RUN ./configure --prefix=/usr --enable-freetype-config
RUN make -j5 install
RUN make -j$(nproc) install

#=================================
WORKDIR /polytracker/the_klondike
WORKDIR $build_dir

#zlib
RUN wget https://www.zlib.net/zlib-1.2.11.tar.gz
# zlib
RUN wget https://www.zlib.net/fossils/zlib-1.2.11.tar.gz
RUN tar -xzvf zlib-1.2.11.tar.gz
WORKDIR zlib-1.2.11
RUN ./configure --prefix=/usr && make -j$(nproc) test && make -j$(nproc) install
WORKDIR $build_dir/zlib-1.2.11
RUN ./configure --prefix=/usr && \
make -j$(nproc) test && \
make -j$(nproc) install

#=================================
WORKDIR /polytracker/the_klondike
WORKDIR $build_dir

#Libxml2
# Libxml2

RUN wget http://xmlsoft.org/sources/libxml2-2.9.10.tar.gz
RUN tar -xvf libxml2-2.9.10.tar.gz
WORKDIR libxml2-2.9.10
RUN wget http://xmlsoft.org/sources/libxml2-2.9.11.tar.gz
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was getting a Python macro error with previous version

RUN tar -xvf libxml2-2.9.11.tar.gz
WORKDIR $build_dir/libxml2-2.9.11
RUN ./configure --disable-dependency-tracking --prefix=/usr --with-python=/usr/bin/python3
RUN make -j5 install
RUN make install

RUN make -j$(nproc) install

#=================================
WORKDIR /polytracker/the_klondike
WORKDIR $build_dir

##Fontconfig (depends on FreeType), note that the linux from scratch version is broken
#The gitlab version is up to date, and has a PR merged from a year ago with the bug fix
#https://gitlab.freedesktop.org/fontconfig/fontconfig/merge_requests/2/diffs?commit_id=8208f99fa1676c42bfd8d74de3e9dac5366c150c
# Fontconfig (depends on FreeType), note that the linux from scratch version is broken
# The gitlab version is up to date, and has a PR merged from a year ago with the bug fix
# https://gitlab.freedesktop.org/fontconfig/fontconfig/merge_requests/2/diffs?commit_id=8208f99fa1676c42bfd8d74de3e9dac5366c150c

RUN git clone https://gitlab.freedesktop.org/fontconfig/fontconfig.git

WORKDIR fontconfig
WORKDIR $build_dir/fontconfig
RUN ./autogen.sh --sysconfdir=/etc --prefix=/usr --enable-libxml2 --mandir=/usr/share/man
RUN make -j5 install
RUN make -j$(nproc) install

#=================================
WORKDIR /polytracker/the_klondike

RUN wget https://dl.xpdfreader.com/xpdf-4.03.tar.gz
RUN tar -xvf xpdf-4.03.tar.gz
WORKDIR xpdf-4.03
RUN mkdir build
WORKDIR build
RUN cmake -DCMAKE_BUILD_TYPE=Release ..
RUN make -j5 install
# build, and record the build with Blight
WORKDIR $build_dir/build
RUN polytracker build cmake -S .. -B . -DCMAKE_BUILD_TYPE=Release
RUN polytracker build make -j$(nproc) install

#=================================
WORKDIR xpdf

#Extract and instrument pdftotext, other poppler tools should work the same
RUN get-bc -b pdftotext
RUN ${CXX} --lower-bitcode -i pdftotext.bc -o pdftotext_track --libs /build_artifacts/libfofi.a /build_artifacts/libgoo.a /build_artifacts/libsplash.a freetype fontconfig pthread --lists freetype fontconfig xml2 libz

RUN get-bc -b pdfinfo
RUN ${CXX} --lower-bitcode -i pdfinfo.bc -o pdfinfo_track --libs /build_artifacts/libfofi.a /build_artifacts/libgoo.a /build_artifacts/libsplash.a freetype fontconfig pthread --lists freetype fontconfig xml2 libz

# Extract and instrument each tool of interest.
# This should make three instrumented binaries (originals live in the xpdf/ directory):
# pdftops.instrumented, pdftotext.instrumented, and pdfinfo.instrumented
# These commands are split up for timing / debugging purposes but you could
# run them all as one big instrument-targets as well.
RUN polytracker instrument-targets --taint --ftrace pdftotext --ignore-lists freetype fontconfig xml2 libz

RUN get-bc -b pdftops
RUN ${CXX} --lower-bitcode -i pdtops.bc -o pdftops_track --libs /build_artifacts/libfofi.a /build_artifacts/libgoo.a /build_artifacts/libsplash.a freetype fontconfig pthread --lists freetype fontconfig xml2 libz
RUN polytracker instrument-targets --taint --ftrace pdfinfo --ignore-lists freetype fontconfig xml2 libz

# Note, the /workdir directory is intended to be mounted at runtime
#VOLUME ["/workdir"]
#WORKDIR /workdir
RUN polytracker instrument-targets --taint --ftrace pdftops --ignore-lists freetype fontconfig xml2 libz
Loading