Skip to content

Commit

Permalink
Merge branch 'bleeding-jumbo' of git://github.com/magnumripper/JohnTh…
Browse files Browse the repository at this point in the history 
…eRipper into bleeding-jumbo
  • Loading branch information
@jfoug
jfoug committed Aug 3, 2015
2 parents 8cb4bc2 + 939d6ce commit 26a1e93
Show file tree
Hide file tree
Showing 28 changed files with 1,068 additions and 334 deletions.
14 changes: 8 additions & 6 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,17 +1,21 @@
*.diff
*.log
*.o
*.orig
*.patch
*.pot
*.rec
*.rej
*~

*.o
*.exe
*.dSYM
.DS_Store

Makefile
john-local.conf

run/SIPdump
run/base64conv
run/calc_stat
run/cprepair
Expand All @@ -23,7 +27,6 @@ run/gpg2john
run/hccap2john
run/john
run/john.local.conf
john-local.conf
run/keepass2john
run/kernels/
run/keychain2john
Expand All @@ -39,7 +42,6 @@ run/pwsafe2john
run/racf2john
run/rar2john
run/raw2dyna
run/SIPdump
run/ssh2john
run/tgtsnarf
run/to_dyna
Expand All @@ -54,29 +56,29 @@ run/wpapcap2john
run/zip2john

src/.gdbinit
src/Unicode
src/aes/aes.a
src/arch.h
src/autoconfig-stamp-h
src/autoconfig-stamp-h-in
src/autoconfig.h
src/autom4te.cache/
src/config*.cache
src/config.status
src/configure.scan
src/config*.cache
src/dyna-comp.dic
src/dyna-comp.in
src/dyna-comp.pot
src/dyna-comp.rec
src/dynamic_big_crypt.c
src/fmt_externs.h
src/fmt_registers.h
src/generic.h
src/john_build_rule.h
src/memdbg_defines.h
src/para-bench
src/stamp-h1
src/Unicode
src/version.h
src/version.h.new
src/dynamic_big_crypt.c

test/
15 changes: 13 additions & 2 deletions doc/README.coding-style
View file
Original file line number Diff line number Diff line change
Expand Up @@ -208,14 +208,25 @@ We use names prefixed by crk_ for global functions in cracker.c, ldr_ for
ones from loader.c and so on.


Chapter 5: Functions
Chapter 5: Declaration

5.1 Declartion
5.1 Functions declartion

In function prototypes, include parameter names with their data types.
Although this is not required by the C language, it is preferred in Linux
because it is a simple way to add valuable information for the reader.

5.2 Variables declaration

Add a blank line after variables declaration. E.g:

void function(void)
{
unsigned char master[32];

sevenzip_kdf((unsigned char*)saved_key[index], master);
}


Chapter 6: Commenting

Expand Down
71 changes: 71 additions & 0 deletions src/CircleCI-MinGW.sh
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
#!/bin/bash
#
# now does mingw-64 build, and linux-64 no-sse2 build (and tests both)

mkdir -p $HOME/bin
cat >$HOME/bin/mingw64 << 'EOF'
Expand Down Expand Up @@ -29,9 +31,18 @@ export PATH="$HOME/bin:$PATH"
# umount /proc/sys/fs/binfmt_misc

mingw64 ./configure --host=x86_64-w64-mingw32
if [ "x$?" != "x0" ] ; then exit 1 ; fi
mingw64 make -sj4
if [ "x$?" != "x0" ] ; then exit 1 ; fi
mv ../run/john ../run/john.exe

# the mingw build does not name many exe files correctly. Fix that. Also strip all exe files for distro.
cd ../run
for f in `ls -l | grep wxr | grep -v [\.][epr] | cut -c 46-` ; do mv $f $f.exe ; done
for f in benchmark-unify mailer makechr relbench ; do mv $f.exe $f ; done
for f in *.exe ; do x86_64-w64-mingw32-strip $f ; done
cd ../src

basepath="/usr/x86_64-w64-mingw32/sys-root/mingw/bin"

find $basepath | grep "dll$"
Expand All @@ -56,4 +67,64 @@ cd /base/JohnTheRipper/run
export WINEDEBUG=-all # suppress wine warnings
/usr/bin/wine john.exe --list=build-info
/usr/bin/wine john.exe --test=0
if [ "x$?" != "x0" ] ; then exit 1 ; fi
/usr/bin/wine john.exe --test=0 --format=dynamic-all
if [ "x$?" != "x0" ] ; then exit 1 ; fi

# now build a non-SIMD 64 bit exe and test it
dnf install -y openssl openssl-devel zlib-devel gmp-devel libpcap-devel
echo ""
echo ""
echo ""
echo ""
echo '******************************************************************************'
echo "now testing a NON-SIMD build"
echo '******************************************************************************'
echo ""
cd /base/JohnTheRipper/src
make -s distclean
CPPFLAGS="-mno-sse2" ./configure
if [ "x$?" != "x0" ] ; then exit 1 ; fi
make -sj4
if [ "x$?" != "x0" ] ; then exit 1 ; fi
../run/john --list=build-info
../run/john -test=0
if [ "x$?" != "x0" ] ; then exit 1 ; fi
../run/john -test=0 -form=dynamic-all
if [ "x$?" != "x0" ] ; then exit 1 ; fi

# now build a non-SIMD 32 bit exe and test it
dnf install -y glibc-headers.i686 glibc.i686 glibc-devel.i686 libgcc.i686 openssl-devel.i686 gmp-devel.i686 libpcap-devel.i686
echo ""
echo ""
echo ""
echo ""
echo '******************************************************************************'
echo "now testing a 32 bit NON-SIMD build"
echo '******************************************************************************'
echo ""
make -s distclean
JOHN_CFLAGS=-m32 JOHN_ASFLAGS=-m32 JOHN_LDFLAGS=-m32 make -f Makefile.legacy -sj4 linux-x86-any
# do NOT exit on error from make. We expect an error in the libpcap stuff
../run/john --list=build-info
../run/john -test=0
if [ "x$?" != "x0" ] ; then exit 1 ; fi
../run/john -test=0 -form=dynamic-all
if [ "x$?" != "x0" ] ; then exit 1 ; fi

# now build a 32 bit SSE2 exe and test it
echo ""
echo ""
echo ""
echo ""
echo '******************************************************************************'
echo "now testing a 32 bit SSE2 build"
echo '******************************************************************************'
echo ""
make -f Makefile.legacy -s clean
JOHN_CFLAGS=-m32 JOHN_ASFLAGS=-m32 JOHN_LDFLAGS=-m32 make -f Makefile.legacy -sj4 linux-x86-sse2
# do NOT exit on error from make. We expect an error in the libpcap stuff
../run/john --list=build-info
../run/john -test=0
if [ "x$?" != "x0" ] ; then exit 1 ; fi
../run/john -test=0 -form=dynamic-all
4 changes: 2 additions & 2 deletions src/Makefile.legacy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ PROJ_WIN32 = find_version ../run/john.exe \
../run/racf2john.exe ../run/hccap2john.exe \
../run/raw2dyna.exe ../run/keepass2john.exe \
../run/pfx2john.exe \
../run/kwallet2john.exe ../run/dmg2john.exe ../run/putty2john.exe \
../run/dmg2john.exe ../run/putty2john.exe \
john.local.conf ../run/gpg2john.exe ../run/base64conv.exe
PROJ_WIN32_MINGW = find_version ../run/john-mingw.exe \
../run/unshadow.exe ../run/unafs.exe ../run/unique.exe \
Expand All @@ -245,7 +245,7 @@ PROJ_WIN32_MINGW = find_version ../run/john-mingw.exe \
../run/racf2john.exe ../run/hccap2john.exe \
../run/raw2dyna.exe ../run/keepass2john.exe \
../run/pfx2john.exe \
../run/kwallet2john.exe ../run/putty2john.exe john.local.conf \
../run/putty2john.exe john.local.conf \
../run/gpg2john.exe ../run/base64conv.exe
PROJ_PCAP = ../run/SIPdump ../run/vncpcap2john
default:
Expand Down
2 changes: 2 additions & 0 deletions src/SybaseASE_fmt_plug.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -145,8 +145,10 @@ static void init(struct fmt_main *self)

static void done(void)
{
#ifdef SIMD_COEF_32
MEM_FREE(last_len);
MEM_FREE(NULL_LIMB);
#endif
MEM_FREE(crypt_out);
MEM_FREE(prep_key);
}
Expand Down
6 changes: 5 additions & 1 deletion src/cryptsha512_fmt_plug.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,11 @@ john_register_one(&fmt_cryptsha512);
#ifdef SIMD_COEF_64
#define ALGORITHM_NAME SHA512_ALGORITHM_NAME
#else
#define ALGORITHM_NAME "64/" ARCH_BITS_STR " " SHA2_LIB
#if ARCH_BITS >= 64
#define ALGORITHM_NAME "64/" ARCH_BITS_STR " " SHA2_LIB
#else
#define ALGORITHM_NAME "32/" ARCH_BITS_STR " " SHA2_LIB
#endif
#endif

// 79 is max length we can do in 2 SIMD limbs, so just make it 79 always.
Expand Down
9 changes: 3 additions & 6 deletions src/dynamic_big_crypt_hash.cin
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,17 +82,14 @@ static void inline Do#{HASH}_sse_crypt_only(void *in, uint32_t len[#{HASH}_LOOPS
loops[i] = Do_FixBufferLen#{BITS}(cp, len[i], #{BE_HASH});
cp += 64*4;
}
cp = (unsigned char*)in;
bMore = 1;
cnt = 1;
cp = (unsigned char*)in; bMore = 1; cnt = 1;
while (bMore) {
#{SSEBody}(cp, out, out, SSEi_FLAT_IN#{SSEFLAGS}|SSEi_OUTPUT_AS_#{SSE_ONLY_LIMBS}INP_FMT|SSEi_#{SSE_LIMBS}BUF_INPUT_FIRST_BLK|(cnt==1?0:SSEi_RELOAD));
#{SSEBody}(cp, a, a, SSEi_FLAT_IN#{SSEFLAGS}|SSEi_OUTPUT_AS_#{SSE_ONLY_LIMBS}INP_FMT|SSEi_#{SSE_LIMBS}BUF_INPUT_FIRST_BLK|(cnt==1?0:SSEi_RELOAD));
bMore = 0;
for (i = 0; i < #{HASH}_LOOPS; ++i) {
if (cnt == loops[i]) {
uint32_t offx = ((i/SIMD_COEF_#{BITS})*(#{BIN_SZ}/sizeof(ARCH_WORD_#{BITS}))*SIMD_COEF_#{BITS})+(i&(SIMD_COEF_#{BITS}-1));
// only 16 bytes in the 'final'
for (j = 0; j < 16/sizeof(ARCH_WORD_#{BITS}); ++j) {
for (j = 0; j < #{BIN_SZ}/sizeof(ARCH_WORD_#{BITS}); ++j) {
((ARCH_WORD_#{BITS}*)out)[(i*#{SSE_LIMBS})+j] = #{JSWAPH}a[(j*SIMD_COEF_#{BITS})+offx]#{JSWAPT}
}
} else if (cnt < loops[i])
Expand Down
2 changes: 1 addition & 1 deletion src/dynamic_fmt.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,6 @@ static union SIMD_crypt {
} *crypt_key, *crypt_key2;
static unsigned int (*total_len)[SIMD_COEF_32];
static unsigned int (*total_len2)[SIMD_COEF_32];
BIG_HASH_OUT dynamic_BHO[4];

#define MMX_INP_BUF_SZ (sizeof(input_buf[0]) *BLOCK_LOOPS)
#define MMX_INP_BUF2_SZ (sizeof(input_buf2[0])*BLOCK_LOOPS)
Expand All @@ -231,6 +230,7 @@ MD5_IN *input_buf_X86;
MD5_IN *input_buf2_X86;
unsigned int *total_len_X86;
unsigned int *total_len2_X86;
BIG_HASH_OUT dynamic_BHO[4];

static int keys_dirty;
// We store the salt here
Expand Down
2 changes: 1 addition & 1 deletion src/dynamic_types.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ typedef struct private_subformat_data
#ifdef SIMD_COEF_32
#define NON_OMP_MAX (SIMD_COEF_32*3*4*5*7)
#else
#define NON_OMP_MAX 1
#define NON_OMP_MAX (1<<MD5_X2)
#endif
#define OMP_MAX (NON_OMP_MAX*OMP_SCALE)

Expand Down
6 changes: 3 additions & 3 deletions src/inc.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -427,9 +427,9 @@ void do_incremental_crack(struct db_main *db, char *mode)
"DefaultIncrementalLM")))
mode = "LM_ASCII";
} else if (db->format->params.label &&
(!strcmp(db->format->params.label, "lm-opencl") ||
!strcmp(db->format->params.label, "netlm") ||
!strcmp(db->format->params.label, "nethalflm"))) {
(!strcasecmp(db->format->params.label, "lm-opencl") ||
!strcasecmp(db->format->params.label, "netlm") ||
!strcasecmp(db->format->params.label, "nethalflm"))) {
if (!(mode = cfg_get_param(SECTION_OPTIONS, NULL,
"DefaultIncrementalLM")))
mode = "LM_ASCII";
Expand Down
Loading

21 comments on commit 26a1e93

@jfoug
Copy link
Collaborator Author

@jfoug jfoug commented on 26a1e93 Aug 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@magnumripper sorry, I think this was a missed --rebase on a pull. I can not figure out how to purge this darn thing.

@frank-dittrich
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jfoug

Apparently, the parent of your commit 8cb4bc2 was magnum's commit c0560a4 from 6 days ago.

Next time, when you notice (before pushing to the upstream repo) that you accidentally merged instead of pull --rebase, you can check your merge commit:

(bleeding-jumbo)JtR $ git show
commit 26a1e93ce395f54527bde4e858a8bd55ca8ce5ed
Merge: 8cb4bc2 939d6ce
Author: jfoug <[email protected]>
Date:   Mon Aug 3 14:57:40 2015 -0500

    Merge branch 'bleeding-jumbo' of git://github.com/magnumripper/JohnTheRipper into bleeding-jumbo

(bleeding-jumbo)JtR $

The Merge: 8cb4bc2 939d6ce line contains both your most recent local commit (HEAD) prior ro merging, and the most recent commit of the upstream repository.
(You can verify this with git show, e.g. git show 8cb4bc2.)

To "undo" the merge in your repository, you can do a 'git reset`.
E.g., if you don't have any changes that are not committed or stashed (so no work should get lost), you can use

(bleeding-jumbo)JtR $ git reset --hard  8cb4bc2
HEAD is now at 8cb4bc2 configure: if fmt_externs.h or fmt_registers.h fail, then bail and reset makefile back to the stub.  #1590
(bleeding-jumbo)JtR $ git status |head -n 3
On branch bleeding-jumbo
Your branch is behind 'origin/bleeding-jumbo' by 46 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)

git log will look like prior to your merge:

commit 8cb4bc2055318afbbc54082f45385bb7374e24b3
Author: jfoug <[email protected]>
Date:   Mon Aug 3 14:54:14 2015 -0500

    configure: if fmt_externs.h or fmt_registers.h fail, then bail and reset makefile back to the stub.  #1590

commit c0560a42f85c437c7b968191e7fb06f1140e8ed9
Author: magnum <[email protected]>
Date:   Tue Jul 28 22:17:32 2015 +0200

    Fix for non-opencl builds.

Then, you can to a git rebase or git pull --rebase and push afterwards.

(Of course, for commit 8cb4bc2 it is now too late.)

You may also consider adding these lines to your globale .gitconfig:

[pull]
    rebase = true
[rebase]
    autostash = true

This will stash your local changes and force a --rebase whenever you pull.

I disabled this, because you loose the nice summary of changed files when pulling.
Instead, I usually do a git fetch and a git status before pulling.

@jfoug
Copy link
Collaborator Author

@jfoug jfoug commented on 26a1e93 Aug 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On this system, I do not have those settings in my .gitconfig for same reason (losing the summary). I simply forgot the --rebase.

@magnumripper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😠

@magnumripper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Losing the summary? We solved that problem half a year ago and correct settings was posted.

@magnumripper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I'm depressed. I'll watch a movie instead.

@frank-dittrich
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Losing the summary? We solved that problem half a year ago and correct settings was posted.

Do you have a pointer? (After you finished watching the movie)

@magnumripper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't remember and I can't find it now. It's probably in the commit comments of last fuckup.

In my john/.git/config I have

[branch "bleeding-jumbo"]
    remote = origin
    merge = refs/heads/bleeding-jumbo
    mergeoptions = --ff-only

In the man page it looks like global option merge.ff = only would have same effect (the above is per branch)

With these settings, you do get a summary. But when you do want to merge you need to say --no-ff.

@magnumripper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

http://aaronbonner.io/post/78444674979/only-allow-git-fast-forward-merges-to-avoid-ugly

$ git config --global merge.ff only

@jfoug
Copy link
Collaborator Author

@jfoug jfoug commented on 26a1e93 Aug 4, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, what do you recommend putting in my ~/.gitconfig (exactly), and what git config ... commands should I use?

@magnumripper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

~/.gitconfig

 [merge]
    (...)
    ff = only

The latter may be added using the command below

$ git config --global merge.ff only

Or you just edit the file, same thing.

@magnumripper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With that setting, and nothing in the way for a clean pull:

$ git pull 
Updating 5cbed9d..b491e49
Fast-forward
 run/john.bash_completion |  14 ++
 src/formats.c            | 648 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------
 src/john.c               |   2 +-
 src/opencl/nt_kernel.cl  |  50 ++----
 src/opencl_nt_fmt_plug.c |   6 +-
 src/options.c            |   4 +
 src/options.h            |   4 +
 src/regex.c              |   4 +-
 8 files changed, 629 insertions(+), 103 deletions(-)

But if you have local commits:

$ git pull
fatal: Not possible to fast-forward, aborting.

$ git pull --rebase 
First, rewinding head to replay your work on top of it...
Applying: Bump nt-opencl to max. length 27.

The downside is when you actually want to merge, you need to say --no-ff

$ git merge origin/topicbranch 
fatal: Not possible to fast-forward, aborting.

$ git merge --no-ff origin/topicbranch
Merge made by the 'recursive' strategy.
 src/opencl/nt_kernel.cl  | 50 ++++++++++++--------------------------------------
 src/opencl_nt_fmt_plug.c |  6 ++++--
 2 files changed, 16 insertions(+), 40 deletions(-)

but that is not the common case anyway.

As always, if you try to push while not in sync:

$ git push
To https://[email protected]/magnumripper/JohnTheRipper.git
 ! [rejected]        bleeding-jumbo -> bleeding-jumbo (non-fast-forward)

When this happens you just git pull --rebase and try again.

@magnumripper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added https://github.com/magnumripper/JohnTheRipper/wiki/Git-settings---tips'n'tricks

@jfoug
Copy link
Collaborator Author

@jfoug jfoug commented on 26a1e93 Aug 5, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The key to getting the summary is to NOT use this:

[pull]
    rebase = true

but to instead use this.

[merge]
    ff = only

This will simply cause a pull to fail if a pull --rebase is required. Good stuff.

@magnumripper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So maybe we should add to the wiki that the former should be dropped (if there)

@jfoug
Copy link
Collaborator Author

@jfoug jfoug commented on 26a1e93 Aug 5, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You still do not get any summary if you do this (but with that in .gitconfig every pull was doing this)

git pull --rebase

It is the rebase process that scrubs the summary. But with the merge=ff we get a normal pull, but that pull will fail with a warning message telling you to rebase if the pull is not able to be done. But most pulls will get a nice summary.

@magnumripper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need a rebase, there is no way to get a summary unless you rebase yourself first, something like this:

$ git fetch
$ git rebase origin/bleeding-jumbo
$ git pull

@magnumripper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No wait, that will not give you a summary either. I'm not sure it can be done.

Anyway, this new setting is much better than the old one. I'm quite happy with it.

@jfoug
Copy link
Collaborator Author

@jfoug jfoug commented on 26a1e93 Aug 6, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I am getting this, and now sure WTF is going on :(

$ git push
fatal: The current branch bleeding-jumbo has multiple upstream branches, refusing to push.

@jfoug
Copy link
Collaborator Author

@jfoug jfoug commented on 26a1e93 Aug 6, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

had some stuff in ~/.gitconfigure that needed cleaned out.

@magnumripper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you didn't have an upstream set for your branch. You could use git push -u origin bleeding-jumbo once and it would be set for future pushes.

Please sign in to comment.