ESP-12E Module: Exception (2) after boot #1675

Jather90 · 2016-02-24T00:52:42Z

Basic Infos

Exception (2) after boot

Hardware

Hardware: ESP-12E
Core Version: 2.1.0-rc2
Windows 10 64-bit

Description

Background: programing ESP-12E worked quite well but randomly gave Exception (28). That's why I updated (read that clue in some other issue that sounded similar) to 2.1.0-rc2 (don't know which version I used before...)
The problem now got even worse! After boot there is now Exception (2) (stack below) and I have no clue what goes wrong.... I'm not sure why, but if I DON'T use "NodeMCU 1.0(ESP-12E Module)" Option to compile and choose "Generic ESP8266Module" with 512K(64 SPIFFS) (and only with this flash size!) everything works fine o.o .... right now I'm not in the mood to check with this configuration for occurance of prior Exception (28) .. I just want to belive they are gone with updating to 2.1.0-rc2! But thats not my problem right now.
Some additional note to flashsize: i tested quite a lot so don't blame me for not knowing every setting in detail, but i got the impression some settings worked as long no WIFI-connection was established. Maybe I'm wrong, but reading stored SSID from EEPROM did not work after changing flashsize settings (I'm not into EEPROM now, i thought that is caused by changing the flash size). After giving SSID via Serial, ESP crashes....

Settings in IDE

Module: NodeMCU 1.0 (ESP-12E Module)
Flash Size: 4MB/1MB
CPU Frequency: 80Mhz
Flash Mode: can't change in this Module
Flash Frequency: can't change in this Module
Upload Using: SERIAL
Reset Method: can't change in this Module

Stack

[hoch() is my Code, looks like this:
368: void hoch(){
369:  Serial.println("called hoch");
370:  wlanHoch=true;
371:  sendDone();
372:}
===========================
void sendDone(){
  server.send ( 200, "text/html",TEXT_DONE); 
  println("send finished");
}
===========================
TEXT_DONE is a simple html in progmem with Text response "DONE!"
const char TEXT_DONE[] PROGMEM = R"=====( .... )=====";
]

Exception (2):
epc1=0x3ffe936c epc2=0x00000000 epc3=0x00000000 excvaddr=0x3ffe936c depc=0x00000000

ctx: cont 
sp: 3fff1270 end: 3fff1630 offset: 01a0

Decoding 35 results
0x4010193e: ppEnqueueRxq at ??:?
0x4010170b: ppProcessTxQ at ??:?
0x40101daa: pp_post at ??:?
0x4022b46c: wifi_softap_set_default_ssid at ??:?
0x4010427b: lmacRecycleMPDU at ??:?
0x401046de: lmacRecycleMPDU at ??:?
0x40107020: printf at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266/libc_replacements.c:206
0x40205553: ESP8266WiFiAPClass::softAP(char const*, char const*, int, int) at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\libraries\ESP8266WiFi\src/ESP8266WiFiAP.cpp:324
0x40205454: ESP8266WiFiAPClass::softAP(char const*, char const*, int, int) at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\libraries\ESP8266WiFi\src/ESP8266WiFiAP.cpp:324
0x4010031d: check_poison at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266\umm_malloc/umm_malloc.c:817
0x40100426: check_poison_block at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266\umm_malloc/umm_malloc.c:851
0x401006d8: malloc at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266\umm_malloc/umm_malloc.c:1658
0x40100537: check_poison_all_blocks at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266\umm_malloc/umm_malloc.c:880
0x401006c2: malloc at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266\umm_malloc/umm_malloc.c:1642
0x40208aa8: operator new(unsigned int) at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266/abi.cpp:84
0x40206ed5: ESP8266WebServer::on(char const*, HTTPMethod, std::function<void ()>, std::function<void ()>) at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\libraries\ESP8266WebServer\src\detail/RequestHandlersImpl.h:53
0x40203604: hoch() at C:\Users\Sven\AppData\Local\Temp\build6345216281220482925.tmp/pages.h:368
0x4020903c: std::_Function_base::_Base_manager<void (*)()>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) at c:\program files (x86)\arduino\hardware\esp8255com\esp8266\tools\xtensa-lx106-elf\xtensa-lx106-elf\include\c++\4.8.2/functional:1934
0x40209028: std::_Function_handler<void (), void (*)()>::_M_invoke(std::_Any_data const&) at c:\program files (x86)\arduino\hardware\esp8255com\esp8266\tools\xtensa-lx106-elf\xtensa-lx106-elf\include\c++\4.8.2/functional:2069
0x40206f4c: ~function at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\libraries\ESP8266WebServer\src\detail/RequestHandlersImpl.h:53
 (inlined by) ESP8266WebServer::on(char const*, HTTPMethod, std::function<void ()>) at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\libraries\ESP8266WebServer\src/ESP8266WebServer.cpp:132
0x4020a830: String::concat(char) at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266/WString.cpp:290
0x40203604: hoch() at C:\Users\Sven\AppData\Local\Temp\build6345216281220482925.tmp/pages.h:368
0x4020903c: std::_Function_base::_Base_manager<void (*)()>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) at c:\program files (x86)\arduino\hardware\esp8255com\esp8266\tools\xtensa-lx106-elf\xtensa-lx106-elf\include\c++\4.8.2/functional:1934
0x40209028: std::_Function_handler<void (), void (*)()>::_M_invoke(std::_Any_data const&) at c:\program files (x86)\arduino\hardware\esp8255com\esp8266\tools\xtensa-lx106-elf\xtensa-lx106-elf\include\c++\4.8.2/functional:2069
0x40203844: readEEPROMAndConnectToWlan() at C:\Program Files (x86)\Arduino/espRollo.ino:246
0x40206f85: ~function at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\libraries\ESP8266WebServer\src\detail/RequestHandlersImpl.h:53
 (inlined by) ESP8266WebServer::on(char const*, std::function<void ()>) at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\libraries\ESP8266WebServer\src/ESP8266WebServer.cpp:128
0x40203604: hoch() at C:\Users\Sven\AppData\Local\Temp\build6345216281220482925.tmp/pages.h:368
0x4020903c: std::_Function_base::_Base_manager<void (*)()>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) at c:\program files (x86)\arduino\hardware\esp8255com\esp8266\tools\xtensa-lx106-elf\xtensa-lx106-elf\include\c++\4.8.2/functional:1934
0x40209028: std::_Function_handler<void (), void (*)()>::_M_invoke(std::_Any_data const&) at c:\program files (x86)\arduino\hardware\esp8255com\esp8266\tools\xtensa-lx106-elf\xtensa-lx106-elf\include\c++\4.8.2/functional:2069
0x40204a3f: ~function at c:\program files (x86)\arduino\hardware\esp8255com\esp8266\tools\xtensa-lx106-elf\xtensa-lx106-elf\include\c++\4.8.2/functional:2174
 (inlined by) setup at C:\Program Files (x86)\Arduino/espRollo.ino:149
0x40203604: hoch() at C:\Users\Sven\AppData\Local\Temp\build6345216281220482925.tmp/pages.h:368
0x4020903c: std::_Function_base::_Base_manager<void (*)()>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) at c:\program files (x86)\arduino\hardware\esp8255com\esp8266\tools\xtensa-lx106-elf\xtensa-lx106-elf\include\c++\4.8.2/functional:1934
0x40209028: std::_Function_handler<void (), void (*)()>::_M_invoke(std::_Any_data const&) at c:\program files (x86)\arduino\hardware\esp8255com\esp8266\tools\xtensa-lx106-elf\xtensa-lx106-elf\include\c++\4.8.2/functional:2069
0x40208bc8: loop_wrapper at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266/core_esp8266_main.cpp:43
0x40100958: cont_norm at C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266/cont.S:109

The text was updated successfully, but these errors were encountered:

Jather90 · 2016-02-24T01:00:33Z

oh nooo.... so right now even Generic Module does not work and instead lead to Exception (2)...... this gives me the hope that even NodeMCU1.0 could work if the exception can be fixed!

Links2004 · 2016-02-24T18:13:48Z

at first test which flash is on your board.
select Flash settings: 512K(64 SPIFFS) + DIO + 40Mhz
then use this example to get the info:
https://github.com/esp8266/Arduino/blob/master/libraries/esp8266/examples/CheckFlashConfig/CheckFlashConfig.ino

when WiFi and EEPROM not working your selection is most likely to big and the code trys to access not existing memory on the Flash.

Jather90 · 2016-02-24T20:36:49Z

Output is:
Flash real id: 001640E0
Flash real size: 4194304

Flash ide size: 524288
Flash ide speed: 40000000
Flash ide mode: DIO
Flash Chip configuration wrong!

Changeing config to:
Flash size: 4M (1M SPIFFS) or 4M (3M SPIFFS) returns "Flash Chip configuration ok"

what do you mean with "selection"?
Compiling with flash size 4M (3M SPIFFS) shows informtation: sketch size 397.251 Bytes (38%) and 38.720 Bytes (47%) for global variables (is it called like this in english? i'm using german IDE).
So i don't think my sketch is too big. Whats confusing to me is that it worked before updating... i didn't change my sketch since updating. should i try to completely remove and reinstall Arduino IDE?

Links2004 · 2016-02-24T21:04:23Z

selection means the flash settings in the IDE, what i mean where if you have real 512KB and select 1MB for example the data will be located outside of the memory range of the Hardware.
but in you case thats not the problem since you have real 4MB.

so you switch from 2.0.0 to 2.1.0-rc2 and now you have the Exception?

i understand german too ;)

uncletammy · 2016-02-24T21:19:01Z

I'm having the same problem and it seems to be related to umm_malloc.

The sketch runs fine up until I call WiFi.begin

I'm running Debian Jessie with a generic ESP12E. I'm flashing with Arduino 1.6.8 using board NodeMCU 1.0 , 80MHZ, 4M (1M SPIFFS).

Sketches that don't connect to Wifi work fine.

Stack

Panic /home/tink/arduino-nightly/hardware/esp8266com/esp8266/cores/esp8266/umm_malloc/umm_malloc.c:851 check_poison_block

ctx: cont 
sp: 3fff0260 end: 3fff0650 offset: 01b0

>>>stack>>>
3fff0410:  00000002 3fff0352 feefeffe feefeffe  
3fff0420:  feefeffe 00000b48 3ffef548 40100445  
3fff0430:  feefeffe feefeffe feefeffe 4010075b  
3fff0440:  feefeffe 00001000 00000484 40100919  
3fff0450:  00000484 00001000 000003fe 401078f4  
3fff0460:  40004b31 3fff0490 0000001c 40227d05  
3fff0470:  40106316 40227ded 3fff124c 000003ff  
3fff0480:  000003fd 3fff0597 3fff124c 000003fd  
3fff0490:  ffffff01 55aa55aa 00000009 0000001c  
3fff04a0:  0000001c 000000a3 00000078 000003ff  
3fff04b0:  402281dc 3fff124c 3fff124c 000000ff  
3fff04c0:  00000001 3fff05b7 40228337 00000008  
3fff04d0:  3fff124c 000000ff 3fff0597 00000000  
3fff04e0:  3fff130c 3fff05f8 00000001 402283c4  
3fff04f0:  3fff0597 3fff124c 00000000 00000004  
3fff0500:  3fff05b7 3fff7584 3fff124c 00000000  
3fff0510:  40228400 3ffe840f 3ffe8400 feefeffe  
3fff0520:  40202b92 3ffe8400 3ffe840f 40202ad2  
3fff0530:  00000000 00000011 3fff0360 3fff0360  
3fff0540:  00000010 00000000 3fff0360 402085fa  
3fff0550:  3ffef300 00000135 00000135 4010020c  
3fff0560:  3ffef0c0 0000000e 3fff0e64 40100aec  
3fff0570:  40228526 3ffef328 3ffef0c0 3fffdad0  
3fff0580:  4022934b 3fff0360 40206304 3ffef3f0  
3fff0590:  ffffff00 41ffffff 61485854 72656b63  
3fff05a0:  63617073 feef0065 feefeffe 00002580  
3fff05b0:  0000001c 68000000 6f6b6361 0021216e  
3fff05c0:  feefeffe 00000001 3ffe8681 4020706c  
3fff05d0:  00000000 0000000e 3ffe83fe 3ffef628  
3fff05e0:  3ffef56c 0000000f 3ffef56c 402053d9  
3fff05f0:  3ffe8680 00fe83ec 3ffef56c 402053d9  
3fff0600:  3ffef310 00000000 00000001 3ffef56c  
3fff0610:  40202907 000003e8 3ffef56c 3ffef628  
3fff0620:  3ffef56c 3ffef310 00000000 40202790  
3fff0630:  3fffdad0 00000000 3ffef621 40205858  
3fff0640:  feefeffe feefeffe 3ffef630 40100b98  
<<<stack<<<

Flash Info

Flash real id:   001640E0
Flash real size: 4194304

Flash ide  size: 4194304
Flash ide speed: 40000000
Flash ide mode:  QIO
Flash Chip configuration ok.

Jather90 · 2016-02-24T21:32:41Z

unfortunatelly i'm not 100% sure which version i used before...it must be released before 12.01.16
i copied the whole foler (arduion)/hardware/esp prior to updating. is there some file where i can find version information?

Jather90 · 2016-02-24T21:39:37Z

i exchange the new (2.1.0-rc2) hardwarefolder with my saved (old and unknown version) and compiled my code. result:
C:\Users\Sven\AppData\Local\Temp\build5520386127692445905.tmp/arduino.ar(umm_malloc.c.o): In function malloc': C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266\umm_malloc/umm_malloc.c:1638: multiple definition ofmalloc'
C:\Users\Sven\AppData\Local\Temp\build5520386127692445905.tmp/arduino.ar(libc_replacements.c.o):C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266/libc_replacements.c:44: first defined here
C:\Users\Sven\AppData\Local\Temp\build5520386127692445905.tmp/arduino.ar(umm_malloc.c.o): In function realloc': C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266\umm_malloc/umm_malloc.c:1687: multiple definition ofrealloc'
C:\Users\Sven\AppData\Local\Temp\build5520386127692445905.tmp/arduino.ar(libc_replacements.c.o):C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266/libc_replacements.c:53: first defined here
C:\Users\Sven\AppData\Local\Temp\build5520386127692445905.tmp/arduino.ar(umm_malloc.c.o): In function free': C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266\umm_malloc/umm_malloc.c:1712: multiple definition offree'
C:\Users\Sven\AppData\Local\Temp\build5520386127692445905.tmp/arduino.ar(libc_replacements.c.o):C:\Program Files (x86)\Arduino\hardware\esp8255com\esp8266\cores\esp8266/libc_replacements.c:48: first defined here
C:\Users\Sven\AppData\Local\Temp\build5520386127692445905.tmp/arduino.ar(umm_malloc.c.o):(.text.check_poison$part$0$constprop$4+0xc): undefined reference to putchar' C:\Users\Sven\AppData\Local\Temp\build5520386127692445905.tmp/arduino.ar(umm_malloc.c.o): In functioncheck_poison$part$0$constprop$4':
umm_malloc.c:(.text.check_poison$part$0$constprop$4+0x3e): undefined reference to `putchar'
collect2.exe: error: ld returned 1 exit status
Fehler beim Kompilieren.

any ideas what went wrong?

Jather90 · 2016-02-24T21:46:50Z

and one other question: could wrong usage of Strings cause Exception (2)? i read about fragmented heap (must read more about it to understand the problem). As I have started my programing experience with java i'm not taht familiar with memory allocation...definitely have to learn more about that and avoid Strings I think...

Links2004 · 2016-02-25T09:37:41Z

copied over is the problem, remove all ESP stuff from the IDE in hardware and in the user folder (APPDATA)
and follow this:
https://github.com/esp8266/Arduino/blob/master/README.md#using-git-version-

for releases the version is noted in platform.txt
but not for staring and so on.
if you use the board manager you see the version in the folder name.
if you use get check the local sha1

note:
when you switch from 2.0.0 to 2.1.0-rcX you need to manually delete the 2.0.0 or it will not work.

Jather90 · 2016-02-25T17:48:26Z

this is what i think updating is:
(0. close Arduino IDE)

save old version: - Go to C:/.../(arduionIDE)/hardware/ and copy folder esp to somewhere else -> for me desktop/OLD. desktop is the usual space to spam folders ;-)
delete appdata/roaming/arduino*
pull newer version (for me commit f28c5be - i also tried with tagged version 2.1.0.rc2) from git (can't pull to (arduino)/hardware/esp due to weired admin settings.... so i pull to desktop/NEW
open cygwin and cd to desktop/NEW/tools and execute "python get.py"
copy desktop/NEW to (arduino)/hardware/

doing this leads to the initial posted exception (2)

doing the same reversed (changing back to old) gives me the error log three posts above.

delete (arduino)/hardware/esp
delete appdata/roaming/arduino*
copy desktop/OLD to (arduino)/hardware/esp
don't execute python, because i executed weeks ago, when i downloaded the old version -> xtensa, etc. exists at OLD/tools

i will try tonight/the next days to complety exchange occurances of String with char-arrays. hope this helps

uncletammy · 2016-02-26T03:51:49Z

@Links2004 , for the record I verified that I'm using version 2.1.0 and I am still having this problem.

Jather90 · 2016-02-26T23:32:57Z

some updates: i started to rebuild my code step by step since i figuered out a blank arduino sketch won't crash my board. So i started copying smal chunks from my previously working code to the blank sketch. Everything was fine, i was happy.... until i reached adding webserver. the board crashed after it had to send contents to a browser that requested some page..... but now, something new! i got Exception (3). nvertheless its frustrating!!
but luckily i saw there were some new commits, especially one looked very interesting: 87a7f8e
and for now it looks great. no exceptions! i'm just testing for 5 minutes, but i'm so happy that all (still!) works that i have to post it! i hope everything will be ok even after adding the rest of the code.

THANK YOU igrr!

(will let this issue open since i haven't tested enough to say it can be closed)

Jather90 · 2016-02-28T14:40:09Z

so i've been testing for some hours now and reactivated my enitre sketch. no errors until now! actually my module is running for more or less 4 hours without restart/reboot/sketchupload and no unwanted resets happend.

uncletammy · 2016-02-28T20:52:21Z

@Links2004, The problem is still unresolved for me. Do you think that our problems are unrelated? If so, that's fine but I would love a response of any sort. ( not that I'm entitled to one. Also, thanks for all the work you've put in to help this community )

Links2004 · 2016-02-28T21:32:51Z

@uncletammy you still have Panic from umm_malloc (heap corrupted)?
can you decode the stack dump to see what you code is doing at least.

uncletammy · 2016-02-28T21:34:56Z

@Links2004, I would love to but I'm not sure how to do that. Give me a few minutes to figure it out and I'll get back to you. Thanks for the response!

Links2004 · 2016-02-28T21:37:33Z

https://github.com/esp8266/Arduino/blob/master/doc/Troubleshooting/stack_dump.md
may you can open a new issues, then we can better rack it.

uncletammy · 2016-02-28T22:12:28Z

@Links2004

There does not appear to be an option to Copy/Paste in ESP Exception Decoder so here's a screenshot

uncletammy · 2016-02-28T22:13:55Z

Also, you can see my sketch at https://gist.github.com/uncletammy/198fec767a8e4803e6af

I've tried the same sketch on two different boards. One is a NodeMCU board and one is a plain generic board that I've wired up. It works on the NodeMCU board but not on the generic board alone. I'm beginning to think it's hardware related. You can see my wiring at the images below.

https://drive.google.com/file/d/0BwFbwCQ9LBckMTk4MWVVSUtGYTA/view?usp=sharing
https://drive.google.com/file/d/0BwFbwCQ9LBckdWhMUVRGNl9UMUk/view?usp=sharing

Jather90 · 2016-02-29T09:43:58Z

hey uncletammy, i'm sorry! i was so happy that my problem is solved that i forgot yours!! just one question in this issue, because then it would be related to mine, other suggestions in your issue: did you merge this ( 87a7f8e ) commit allready?

uncletammy · 2016-03-05T02:25:03Z

@Jather90, that's okay. Thanks for coming back. Unfortunately I have already merged that commit and I'm still having issues :(

tablatronix · 2016-03-06T03:42:37Z

I can reproduce as well, was following other thread, just found this one will read it all and do a cleanup and see.

clovisf · 2016-03-29T22:55:04Z

I was getting the same error; implemented the solution proposed by @Jather90 (change the Flash size to 4M (1M SPIFFS) or 4M (3M SPIFFS)). I got "Flash Chip configuration ok"" in the flash size sketch and finally got my ESP12E to connect to wifi. I am using windows 10 and Arduino IDE 1.6.8 Hourly build 2016/03/08 06:33

igrr added type: troubleshooting waiting for feedback Waiting on additional info. If it's not received, the issue may be closed. labels Feb 27, 2016

Jather90 closed this as completed Feb 28, 2016

uncletammy mentioned this issue Feb 29, 2016

ESP12-E : Panic from umm_malloc ( Inconsistent between similar ESPs ) #1699

Closed

tablatronix mentioned this issue May 6, 2016

2.2.0 Exception (3): #1997

Closed

igrr added a commit that referenced this issue May 10, 2016

Prevent WiFi config corruption (#1997 #1856 #1699 #1675)

f2fb43c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ESP-12E Module: Exception (2) after boot #1675

ESP-12E Module: Exception (2) after boot #1675

Jather90 commented Feb 24, 2016

Jather90 commented Feb 24, 2016

Links2004 commented Feb 24, 2016

Jather90 commented Feb 24, 2016

Links2004 commented Feb 24, 2016

uncletammy commented Feb 24, 2016

Jather90 commented Feb 24, 2016

Jather90 commented Feb 24, 2016

Jather90 commented Feb 24, 2016

Links2004 commented Feb 25, 2016

Jather90 commented Feb 25, 2016

uncletammy commented Feb 26, 2016

Jather90 commented Feb 26, 2016

Jather90 commented Feb 28, 2016

uncletammy commented Feb 28, 2016

Links2004 commented Feb 28, 2016

uncletammy commented Feb 28, 2016

Links2004 commented Feb 28, 2016

uncletammy commented Feb 28, 2016

uncletammy commented Feb 28, 2016

Jather90 commented Feb 29, 2016

uncletammy commented Mar 5, 2016

tablatronix commented Mar 6, 2016

clovisf commented Mar 29, 2016

ESP-12E Module: Exception (2) after boot #1675

ESP-12E Module: Exception (2) after boot #1675

Comments

Jather90 commented Feb 24, 2016

Basic Infos

Hardware

Description

Settings in IDE

Stack

Jather90 commented Feb 24, 2016

Links2004 commented Feb 24, 2016

Jather90 commented Feb 24, 2016

Links2004 commented Feb 24, 2016

uncletammy commented Feb 24, 2016

Stack

Flash Info

Jather90 commented Feb 24, 2016

Jather90 commented Feb 24, 2016

Jather90 commented Feb 24, 2016

Links2004 commented Feb 25, 2016

Jather90 commented Feb 25, 2016

uncletammy commented Feb 26, 2016

Jather90 commented Feb 26, 2016

Jather90 commented Feb 28, 2016

uncletammy commented Feb 28, 2016

Links2004 commented Feb 28, 2016

uncletammy commented Feb 28, 2016

Links2004 commented Feb 28, 2016

uncletammy commented Feb 28, 2016

uncletammy commented Feb 28, 2016

Jather90 commented Feb 29, 2016

uncletammy commented Mar 5, 2016

tablatronix commented Mar 6, 2016

clovisf commented Mar 29, 2016