-
Notifications
You must be signed in to change notification settings - Fork 206
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
use "session.runner.run " to predict with tensors,after many times, the memory grow higher and higher then oom #567
Comments
Are you mixing TF-Java versions? 1.0.0 doesn't run on Java 8, and you really shouldn't mix tensorflow-core-api 0.4.2 with tensorflow-core-native 1.0.0-rc.2. |
@Craigacp |
We dropped support for Java 8 with TF-Java 0.5.0. That version and 1.0.0-rc.2 require Java 11. You can probably compile the v1.0.0 tag with Java 8 by changing the pom file and removing the error-prone & spotless checks, but we're not supporting it anymore. |
@Craigacp hi, i have tried jdk 11, and tensorflow-core-api / native: 1.0.0-rc.2 jdk: openjdk version "11.0.23" 2024-04-16
|
Is the Java heap expanding and triggering an OOM, or is the native heap expanding? How are the Java GC parameters set? |
@Craigacp yes, there has oom, |
@Craigacp java heap is ok, only 3GB, but total cost 25.1G by tensorflow graph and runtime tensorflow netive memory |
@Craigacp after shell pmap -x "mypid" |
@Craigacp we find the memory cost by [ anon ], |
Ok, I'll see if I can replicate it and run it down. |
@Craigacp thanks so much , we will not close session and graph, so i will session.runner.feed/fetch many times,even though I have closed the tensor generated each time, I am unsure if there is a cache in the session.runner, causing memory to gradually be consumed as the number of requests increases.
|
@Craigacp hi, Any progress? |
@hanfengatonline Could you try with Edit: apologies, I was fearing a native memory leak, but that doesn't seem to be the case (see #588 for more context). |
I ported your code over to Java so I could test it and I couldn't see any leak when running it. My memory usage was pretty solid at 2.7GB on a 2023 M3 Max MBP, using Java 17. I had to move the runner creation inside the loop so it didn't crash, runners should be short lived objects for a single execution of the TF graph. import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.tensorflow.SavedModelBundle;
import org.tensorflow.Session;
import org.tensorflow.Tensor;
import org.tensorflow.ndarray.Shape;
import org.tensorflow.ndarray.impl.buffer.raw.RawDataBufferFactory;
import org.tensorflow.proto.ConfigProto;
import org.tensorflow.proto.RunOptions;
import org.tensorflow.types.TFloat32;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
public class MemLeakTest {
public static void main(String[] args) throws JsonProcessingException {
var kernel = loadTensorflowModel("~/Downloads/test_model");
var json = "{\"serving_default_trained_sub_model_score\":[\"NaN\"],\"serving_default_inner_pay_gap_day_seq\":[\"22\",\"95\",\"114\",\"115\",\"122\",\"137\",\"253\"],\"serving_default_input_profile\":[\"6.0\",\"1\",\"6\",\"0.16634308\",\"0.44211066\",\"0.27592358\",\"0.046576172\",\"0.07684306\",\"0.1421662\",\"0\",\"0.622222222222\",\"58\",\"0.513274336283\",\"0.0647766602711\",\"0.0566966144199\",\"0.34084162\",\"0.0806018806897\",\"0.016667\",\"0.071429\",\"3487\",\"1\",\"1\",\"-2.9616807\",\"-1.6372639\",\"-0.7149122\",\"-1.2260734\",\"0.7172737\",\"-0.85318327\",\"-0.9391162\",\"0.093568556\",\"3.7465565\",\"-1.4456742\",\"0.22602487\",\"-1.0320054\",\"1.1451173\",\"0.2524485\",\"-3.0838134\",\"0.588348405415\",\"378\",\"0\",\"0.75\",\"6\",\"0\",\"2.13522742951\",\"6.38635992548\",\"0\",\"2.44948974278\",\"29.1197551762\",\"8600000000\",\"0\",\"1\",\"0\",\"0\",\"0\",\"1\",\"1\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0.118677705526\",\"0.0374854803085\",\"0.229822456837\",\"0.981258451939\",\"0.708709359169\",\"0\",\"0.648167073727\",\"0.0640747249126\",\"0.940500199795\",\"0.184827804565\",\"0.212323963642\",\"9\",\"3\",\"0\",\"1\",\"0.129026964801\",\"4053.47396349\",\"1708\",\"0.0596125186289\",\"0.0298062593145\",\"0.740323610815\",\"0\",\"0\",\"0\",\"0\",\"0.910581222057\",\"0.421034703002\",\"3\"],\"serving_default_input_product_emb\":[\"1.0954206\",\"0.0007562159\",\"1.4599175\",\"-0.00014710308\",\"-0.0010644994\",\"0.0013906389\",\"-3.69711e-05\",\"-0.001670132\",\"-0.0022175913\",\"0.37936488\",\"0.5284309\",\"0.56634104\",\"1.1236839\",\"0.0017568478\",\"-0.00023513066\",\"0.0010139837\",\"1.1543441\",\"-0.39356884\",\"0.61944026\",\"-5.8332167e-05\"],\"serving_default_input_author_emb\":[\"-3.677482\",\"3.0068984\",\"-2.033637\",\"-3.3155186\",\"-2.1675327\",\"1.2558748\",\"-0.5111591\",\"2.5993257\",\"-0.82787174\",\"-2.3015835\",\"4.0934296\",\"1.0325292\",\"-6.5000277\",\"0.166066\",\"-6.146536\",\"-2.5257614\",\"-0.5164101\",\"3.0859792\",\"-3.6053128\",\"-1.613338\",\"-6.921868\",\"1.3777921\",\"0.15768814\",\"2.6957407\",\"3.7368975\",\"2.6988764\",\"5.4564147\",\"-1.459381\",\"4.033972\",\"-1.4000022\",\"-3.3602858\",\"2.8333883\",\"-5.292586\",\"-0.12055129\",\"-4.8627324\",\"-1.8387526\",\"0.7533025\",\"-3.0386224\",\"8.36378\",\"-2.9174833\",\"-3.233302\",\"-2.0123801\",\"-0.30617282\",\"-0.4304892\",\"6.454887\",\"2.0576859\"],\"serving_default_input_ecom\":[\"0.666666666667\",\"0.666666666667\",\"3.46153846154\",\"0.25\",\"1\",\"0.125\",\"6990\",\"0.110045813064\",\"19980\",\"2.625\",\"0.0967741935484\",\"0.5\",\"12539.5\",\"-1\",\"0\",\"1189.7\",\"-100\",\"43\",\"0\",\"0\",\"0.458333333333\",\"8580.33333333\",\"0.458333333333\",\"3.75\",\"1.63636363636\",\"0.2\",\"9990\",\"0.208333333333\",\"2530.48\",\"1895.29\",\"16900\",\"169\",\"0\",\"0.125\",\"0.1875\",\"0.75\",\"7590\",\"0.63852446987\",\"0.97821619573\",\"0\",\"0.030303030303\",\"0.0330578512397\",\"4.84106818771\",\"4.875\",\"293\",\"0\",\"0\",\"49900\",\"7900\",\"38900\",\"29800\",\"4999\",\"4999\",\"7900\",\"1990\",\"0\",\"0\",\"1990\",\"0.25101561759\",\"0.0606060606061\",\"0.204103887724\",\"54.78\",\"6\",\"8\",\"5\",\"233\",\"109\",\"14\",\"269\",\"0.4\",\"0.657142857143\",\"0.287769784173\",\"0\",\"5\",\"0.775\",\"0.35\",\"32\",\"0.28164556962\",\"0\",\"5\",\"0.359550561798\",\"74\",\"0.831460674157\",\"89\",\"1\",\"0.875\",\"0.439873417722\",\"9.920354\",\"9\",\"1121\",\"9.461538\",\"5\",\"123\",\"3.23\",\"5.67\",\"1.4444444\",\"3.7368073\",\"600\",\"0\",\"0\",\"1\",\"2\",\"1\",\"300\",\"1\",\"300\",\"0.0756034\",\"0\",\"0\",\"0\",\"39900\",\"8991.063\",\"0\",\"2.2173562\",\"0\",\"0.1\",\"0\",\"0.1\",\"13800\",\"19800\",\"1\",\"1990\",\"14990\",\"7584.1665\",\"2\",\"9990\",\"1090\",\"29700\",\"0\",\"7.727525\",\"990\",\"12591.786\",\"2945\",\"-0.9216966\",\"14\",\"0.3887809\",\"-0.034732345\",\"9990\",\"13067535\",\"2679314.5\",\"417107.03\",\"0\",\"0.1113463\",\"0.9142857\",\"2.0270755\",\"1090\",\"0.33333334\",\"0.027777778\",\"0.9166667\",\"3369.009\",\"0.33333334\",\"3065\",\"4490\",\"1\",\"1.1774458\",\"116.37\",\"0.193786031376\",\"4\",\"0.8\",\"116.37\",\"4\",\"0.309203450775\",\"0.352112676056\",\"4\",\"4\",\"149.9\",\"4\",\"0.946666666667\",\"1.70359701196\",\"2.67804173859\",\"28.8862666667\",\"0\",\"2.98689723224\",\"2.97145003934\",\"4.14914227748\",\"0\",\"3\",\"5.74193548387\",\"37\",\"7.3164556962\",\"7.3164556962\",\"44\",\"1.44357080233\",\"0.785714285714\",\"0.272977369149\",\"195198.4\",\"0.161902313592\",\"0.814814814815\",\"0.169973818382\",\"0.822101415924\",\"0.822694820496\",\"1.09660780582\",\"588214.666667\",\"1.36643513432\",\"0.928571428571\",\"1\",\"0.067531948026\",\"0\",\"0.851851851852\",\"0.925925925926\",\"0\",\"0.760891205522\",\"0.394536180641\",\"1.17341440601\",\"1.04531870811\",\"1.03985341166\",\"0.594201949522\",\"0.791215315967\",\"0.857142857143\",\"0.444444444444\",\"0.879436130573\",\"0.15173594712\",\"0.597501416909\",\"0.629928315412\",\"0.687306501548\",\"0.916517323775\",\"0.856118791603\",\"0.31541218638\",\"0.0034784903904\",\"0.737288135593\",\"0.911111111111\",\"0\",\"0.573033707865\",\"0.094674556213\",\"1\",\"72\",\"2\",\"0.0330578512397\",\"3\",\"22.4646332645\",\"38.4798068182\",\"-1.41666666667\",\"-527.033333333\",\"-50.3083333333\",\"51179\",\"238319\",\"0.96\",\"80\",\"239709\",\"0.205882352941\",\"0.0909090909091\",\"0.163742690058\",\"0\",\"0.250975292588\",\"769\",\"0.0469387755102\",\"0.165178571429\",\"2\",\"0\",\"0.186200378072\",\"60\",\"0.0567107750473\",\"0.034703196347\",\"1095\",\"0.4\",\"15104\",\"0.177966101695\",\"18\",\"0.03\",\"1\",\"0.00416666666667\",\"0.0436331255565\",\"2246\",\"0.250396196513\",\"3\",\"0.032967032967\",\"1260\",\"355\",\"0.144543973941\",\"39\",\"1\",\"85\",\"109\",\"0.988235294118\",\"0.0652617642191\",\"0.862595419847\",\"0.486921529175\",\"0\",\"0.049969897652\",\"0.142857142857\",\"2\",\"0.187878787879\",\"0.1231281198\",\"5.9113955\",\"6.3473682\",\"0.36491227\",\"1.4257593\",\"0.4\",\"0\",\"3287785\",\"0.051282052\",\"1489158.6\",\"0.02631579\",\"0.11392166\",\"0.8974359\",\"15153.632\",\"3785.1487\",\"100\",\"3.0971355\",\"0.16866158\",\"0.15\",\"629.5\",\"167.5\",\"0.013411666\",\"0.725\",\"-0.15372944\",\"1.236068\",\"0.075\",\"0.525\",\"0.26585367\",\"1\",\"-1.7106744\",\"0.024163697\",\"0.0051227\",\"0.005735701\",\"0.044593524\",\"0.010041665\",\"0.012656357\",\"0.01228734\",\"0.00412808\",\"0.026462212\",\"-0.113751255\",\"0.020095252\",\"-0.117166795\",\"0.2551187\",\"-0.004284018\",\"-0.0887768\",\"0.083127946\",\"-0.06744456\",\"-0.13762826\",\"0.12308418\",\"0.06441258\",\"-0.09370874\",\"-0.27996\",\"0.20767967\",\"-0.02373123\",\"-0.017548386\",\"-0.17260143\",\"-0.04338026\",\"0.027628133\",\"-0.039716743\",\"0.010191949\",\"-0.1373482\",\"-0.3102747\",\"0.030340211\",\"0.22555368\",\"-0.03646872\",\"2\",\"1\",\"0.0558659217877\",\"0.0111111111111\",\"7\",\"2146\",\"50\",\"0.295857988166\",\"0.530474040632\",\"1\",\"0\",\"0\",\"30\",\"1\",\"60\",\"50\",\"3\",\"95.9909917355\",\"0.153846153846\",\"19\",\"4\",\"1691062874\",\"498\",\"0.338444687842\",\"8\",\"1\",\"48\",\"8\",\"0.923797642332\",\"56\",\"0.225563909774\",\"0.00501253132832\",\"1563725\",\"0.648879510848\",\"0.624060150376\",\"91\",\"471911\",\"0.185463659148\",\"13\",\"0.0325814536341\",\"1000\",\"0.000414957560216\",\"2\",\"1\",\"0.8675\",\"0.599259895701\",\"12150\",\"127917\",\"0.435897435897\",\"0.272686985424\",\"0.418803418803\",\"498784\",\"65900\",\"210339\",\"65900\",\"17\",\"0.145299145299\",\"6000\",\"0.0228310502283\",\"0.230949463186\",\"0.525862068966\",\"18\",\"0.015625\"],\"serving_default_input_type\":[\"1.0\",\"1.0\",\"1.0\",\"1.0\",\"1.0\",\"1.0\",\"1.0\",\"1.0\",\"1.0\",\"1.0\"],\"serving_default_input_aed\":[\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"2\",\"-1\",\"1\",\"1\",\"-1\",\"1\",\"1\",\"1\",\"305\",\"1\",\"1\",\"NaN\",\"NaN\",\"1\",\"NaN\",\"NaN\",\"1\",\"1\",\"NaN\",\"1\",\"1\",\"2\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"1\",\"NaN\",\"NaN\",\"NaN\",\"1\",\"NaN\",\"NaN\",\"-1\",\"1\",\"NaN\",\"1\",\"1\",\"NaN\",\"NaN\",\"11\",\"-1\",\"1\",\"305\",\"NaN\",\"294\",\"11\",\"1\",\"1\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"-1\",\"1\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"1\",\"294\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"1\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"1\",\"1\",\"1\",\"1.00\",\"NaN\",\"1\",\"1\",\"1\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"1\",\"NaN\",\"1.00\",\"NaN\",\"NaN\",\"1\",\"-1\",\"NaN\",\"NaN\",\"NaN\",\"1\",\"1\",\"1\",\"NaN\",\"NaN\",\"NaN\",\"1\",\"1\",\"NaN\",\"1\",\"NaN\",\"NaN\",\"0\",\"NaN\",\"1\",\"294\",\"NaN\",\"1\",\"NaN\",\"1\",\"NaN\",\"2\",\"NaN\",\"NaN\",\"1\",\"1\",\"NaN\",\"NaN\",\"NaN\",\"1\",\"0\",\"0\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"0\",\"NaN\",\"-1\",\"NaN\",\"1\",\"NaN\",\"1.00\",\"NaN\",\"1\",\"NaN\",\"1\",\"NaN\",\"NaN\",\"NaN\",\"1\",\"1\",\"1.00\",\"NaN\",\"294\",\"1\",\"NaN\",\"NaN\",\"3\",\"NaN\",\"2.842829193974446\",\"223.3349502429031\",\"4\",\"16.0\",\"11408.0\",\"NaN\",\"NaN\",\"200.0\",\"1621.0\",\"189954.0\",\"0\",\"NaN\",\"5063.5\",\"NaN\",\"3338\",\"6.0\",\"18.0\",\"2.0\",\"0.0\",\"67.2642105263158\",\"NaN\",\"0.3498\",\"0\",\"NaN\",\"1749.0\",\"8\",\"0.3498\",\"18.0\",\"63\",\"NaN\",\"46\",\"0\",\"1762.1875\",\"291408.0\",\"NaN\",\"5000.0\",\"62\",\"1.0\",\"33983\",\"NaN\",\"NaN\",\"1814\",\"3\",\"0.3793103448275862\",\"NaN\",\"1\",\"625.0\",\"5\",\"50.0\",\"20.0\",\"115.2073732718894\",\"NaN\",\"41.0\",\"0\",\"117\",\"34.98\",\"NaN\",\"44\",\"36.48219178082192\",\"33983.0\",\"1749.0\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"33.0\",\"2754.0\",\"0.5141035714285714\",\"450.0\",\"20672.0\",\"4008.0\",\"5000.0\",\"0\",\"NaN\",\"9.0\",\"2\",\"0.0\",\"280000.0\",\"-26\",\"0.7248720048664955\",\"6\",\"0.0\",\"33.0\",\"8\",\"1.0\",\"352.0\",\"4008.0\",\"0.3793103448275862\",\"0.0\",\"3408.0\",\"5000.0\",\"14.0\",\"0.010539393747960032\",\"23.529411764705884\",\"NaN\",\"0.0\",\"1129\",\"8904.09375\",\"33983\",\"0.0\",\"22.0\",\"7\",\"280000.0\",\"1\",\"58.988195615514336\",\"6\",\"66\",\"4.0\",\"0.3498\",\"1500.0\",\"40\",\"7\",\"10.0\",\"0.6206896551724138\",\"28.0\",\"NaN\",\"16167\",\"507.4246752183107\",\"-8498.64705882353\",\"50\",\"18.0\",\"18.0\",\"61.0\",\"34.98\",\"NaN\",\"0.0\",\"2\",\"0.0\",\"0\"],\"serving_default_inner_pay_trade_type_seq\":[\"1\",\"2\",\"1\",\"1\",\"1\",\"1\",\"1\"],\"serving_default_input_douyin\":[\"0.015364182\",\"0.022161229\",\"0.047867857\",\"0.024159236\",\"0.019388612\",\"0.015773335\",\"0.014519561\",\"0.048452497\",\"0.01395968\",\"0.0146678025\",\"0.039789665\",\"0.05345342\",\"0.015731351\",\"0.017113963\",\"0.018419135\",\"0.01637872\",\"0.01319824\",\"0.07278266\",\"0.019222887\",\"0.009909111\",\"0.013782787\",\"0.011220877\",\"0.01734077\",\"0.03857247\",\"0.010123759\",\"1.0840887\",\"0.05621\",\"7\",\"1.496301\",\"2.1177273\",\"1\",\"2.0536497\",\"1.074602\",\"1.109106\",\"3\",\"0\",\"1.6814331\",\"2.0536497\",\"0\",\"1.0679404\",\"1.2790573\",\"1\",\"0.50041\",\"0.030303031\",\"0.03883495\",\"1.5347259\",\"1.7301531\",\"5\",\"1\",\"6\",\"0\",\"0\",\"0\",\"0\",\"1.0794318\",\"6.9139533\",\"22\",\"2576\",\"154\",\"2361\",\"0\",\"0\",\"94\",\"-3\",\"2\",\"7\",\"3\",\"3\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"1\",\"1\",\"0\",\"0\",\"1\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"13943\",\"3\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"7\",\"0\",\"482590\",\"20\",\"1\",\"15766\",\"6\",\"3\",\"33\",\"54\",\"5\",\"5\",\"1\",\"10\",\"2\",\"2\",\"51772\",\"5\",\"1\",\"0\",\"80075\",\"1941945\",\"3\",\"0\",\"4\",\"690\",\"0\",\"0\",\"0\",\"1\",\"2\",\"5\",\"1\",\"6\",\"0\",\"2\",\"4\",\"3889392\",\"171008\",\"11\",\"1\",\"0\",\"0\",\"361567\",\"3\",\"118\",\"1\",\"13\",\"178\",\"6\",\"1\",\"59\",\"4\",\"1\",\"4\",\"69\",\"3\",\"1\",\"2\",\"6\",\"7\",\"399\",\"18\",\"113\",\"2\",\"0\",\"300\",\"77510467\",\"237\",\"2\",\"0\",\"0\",\"14\",\"11779\",\"44\",\"0\",\"0\",\"0\",\"0\",\"17374482\",\"368362000\",\"2\",\"7\",\"11\",\"3\",\"2\",\"4\",\"6\",\"6\",\"0\",\"22\",\"63\",\"18\",\"2963417\",\"132\",\"2\",\"1\",\"1\",\"18\",\"0\",\"4\",\"41\",\"3421\",\"136\",\"0\",\"0\",\"0\",\"1624397\",\"50\",\"24\",\"147\",\"171008\",\"11\",\"0\",\"0\",\"21\",\"2\",\"58\",\"21240\",\"0\",\"70992\",\"4\",\"-0.238098144531\",\"0.334350585938\",\"-0.365112304688\",\"-0.307006835938\",\"0.0566558837891\",\"0.0533294677734\",\"-0.332641601562\",\"0.180725097656\",\"-0.175842285156\",\"-0.195617675781\",\"-0.0202255249023\",\"-0.0471649169922\",\"-0.0341033935547\",\"0.0243453979492\",\"-0.286743164062\",\"0.139465332031\",\"-0.235412597656\",\"-0.475219726562\",\"-0.447631835938\",\"0.145568847656\",\"-0.167419433594\",\"-0.0851745605469\",\"0.0309524536133\",\"-0.197326660156\",\"0.0363006591797\",\"0.0610198974609\",\"-0.212463378906\",\"0.0268936157227\",\"0.155456542969\",\"-0.0501251220703\",\"-0.125793457031\",\"0.317016601562\",\"-0.202087402344\",\"-0.0448455810547\",\"0.143493652344\",\"-0.165100097656\",\"0.0815734863281\",\"0.158508300781\",\"0.196350097656\",\"0.0628967285156\",\"0.404663085938\",\"-0.0847473144531\",\"0.354125976562\",\"-0.195251464844\",\"0.230651855469\",\"-0.115875244141\",\"-0.148254394531\",\"0.25788942\",\"0.0007679774\",\"0.0013244856\",\"0.0018554225\",\"0.0012003026\",\"0.0015898851\",\"0.002294816\",\"0.001212182\",\"0.0016692934\",\"0.0010517475\",\"0.0014232163\",\"0.006211342\",\"0.4402795\",\"1\",\"0\",\"0\",\"0\",\"0.54418945\",\"-9999\",\"0.6550293\",\"0.353232828237\",\"0.189027950168\",\"0.3780559\",\"0.331337230239\",\"0.371525881499\",\"0.872881355932\",\"0\",\"0\",\"0.955555555556\",\"1\",\"60\",\"14\",\"625\",\"0\",\"90\",\"15\",\"16\",\"13\",\"0\",\"394\",\"1\",\"5.22\",\"0.19934468\",\"0.12409516\",\"-1\",\"-1\",\"-1\",\"-1\",\"NaN\",\"1\",\"4\",\"0\",\"1\",\"0\",\"-1\",\"1\",\"1\",\"4455\",\"0\",\"0\",\"1\",\"1.66666666667\",\"1\",\"11\",\"0\",\"6436\",\"58691\",\"0\",\"0\",\"0\",\"0\",\"1.56\",\"2\",\"1\",\"9\",\"1\",\"1\",\"1\",\"1\",\"2\",\"1\",\"31\",\"45\",\"46\",\"47\",\"48\",\"80\",\"16856\",\"9\",\"2\",\"1\",\"932\",\"1\",\"0.0048016105\",\"0.004252207\",\"0.016223682\",\"0.009352278\",\"0.0048474795\",\"0.052074503\",\"0.012187798\",\"6.605\",\"0\",\"0\",\"2.265\",\"0\",\"228.247\",\"35.374\",\"0\",\"446.656\",\"2798.884\",\"0.308499991894\",\"6.346\",\"89.4156996906\",\"95.547\",\"3\",\"0\",\"0\",\"0\",\"0\",\"0\",\"74.831\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0.053314775\",\"1.0900364e-05\",\"1.0758892e-05\",\"1.0781922e-05\",\"1.0713e-05\",\"1.145243e-05\",\"1.08914e-05\",\"1.0883709e-05\",\"1.1084266e-05\",\"0.021303782\",\"1.0953893e-05\",\"1.1278608e-05\",\"1.0947414e-05\",\"1.1745709e-05\",\"1.1423381e-05\",\"1.0942804e-05\",\"1.0833762e-05\",\"0.06448898\",\"1.0900845e-05\",\"1.087283e-05\",\"1.11587615e-05\",\"1.1526876e-05\",\"1.157721e-05\",\"1.077442e-05\",\"0.018841412\",\"1.0981802e-05\",\"1.1307748e-05\",\"1.0594342e-05\",\"1.1384831e-05\",\"1.2230199e-05\",\"1.0779801e-05\",\"1.1218836e-05\",\"0.26357314\",\"1.0976754e-05\",\"0.055758826\",\"0.07733623\",\"1.1496058e-05\",\"0.02944796\",\"1.1229708e-05\",\"1.0900432e-05\",\"1.0779492e-05\",\"0.24316424\",\"1.0952731e-05\",\"0.034155462\",\"1.1045412e-05\",\"1.1203595e-05\",\"1.0781807e-05\",\"1.126557e-05\",\"1.0977601e-05\",\"1.1021473e-05\",\"1.1124115e-05\",\"1.1364056e-05\",\"0.07425524\",\"1.0867204e-05\",\"0.017593198\",\"1.08396425e-05\",\"1.15695875e-05\",\"1.113451e-05\",\"1.0716846e-05\",\"0.043252926\",\"1.1433921e-05\",\"0.0065541463\",\"0.021663623\"],\"serving_default_input_cate\":[\"269.89\",\"0.80441266\",\"968.99\",\"0.5\",\"0.64850247\",\"4\",\"0.875\",\"2\",\"0.083333336\",\"287.9\",\"0.96359307\",\"0.125\",\"0.5\",\"0.64850247\",\"1546.59\",\"0.6111844\",\"17\",\"0.61707264\",\"5\",\"0.8333333\",\"0.510417\",\"4\",\"0.33333334\",\"0.5\",\"0.7916667\",\"-1\",\"0.337542662116\",\"0.30614334471\",\"0.125\",\"0.125\",\"0.115021975529\",\"0.105519471954\",\"0.0487804878049\",\"0.0406504065041\",\"12.1230982203\",\"-0.007178601\",\"-0.0033580181\",\"0.00018617582\",\"-0.021883385\",\"0.016588738\",\"-0.011529422\",\"0.016623275\",\"0.0012452232\",\"-0.020728054\",\"-0.016857037\",\"0.017310059\",\"-0.013332386\",\"-0.0067435307\",\"0.0010332204\",\"0.02989477\",\"-0.026923006\",\"-0.023780176\",\"-0.0146106435\",\"-0.0014884775\",\"-0.029838799\",\"-0.007271569\",\"0.020200195\",\"0.018829055\",\"0.21428572\",\"0.21875\",\"0.33333334\",\"0.27272728\",\"0.39130434\",\"0.3114754\",\"0.2857143\",\"0\",\"0.153846153846\",\"0.11843003413\",\"0.111604095563\",\"0.153846153846\",\"584.3\",\"0.07692308\",\"0.63825566\",\"0.6462925\",\"0.23076923\",\"3015.3\",\"0.17391305\",\"0.81573594\",\"0.87058204\",\"0.3478261\",\"789.65\",\"43\",\"0.22395833\",\"0.25\",\"1110.95\",\"500.4\",\"15\",\"0.171875\",\"0.1875\",\"0.13333334\",\"0.6152951\",\"0.6483995\",\"0.26666668\",\"1259.53\",\"39.95\",\"0.007915123\",\"0.94383126\",\"0\",\"107390\",\"0.81573594\",\"0.87058204\",\"0.15\",\"0.68032366\",\"0.63054687\",\"0.25\",\"7991.68\",\"3555.65\",\"104\",\"0.27272728\",\"0\",\"0\",\"0.617023\",\"2\",\"2\",\"0.25\",\"0.2857143\",\"0.106777206\",\"0.99574804\",\"1\",\"0\",\"0.125\",\"0\",\"0.7114384\",\"0.71580714\"],\"serving_default_inner_pay_pay_status_seq\":[\"1\",\"1\",\"1\",\"1\",\"1\",\"2\",\"1\"],\"serving_default_input_real_time\":[\"0\",\"19\",\"3\",\"0.6838\",\"0\",\"4.6896095\",\"13.2963332251\",\"0.408649257202\",\"0\",\"0\",\"1\",\"327\",\"0\",\"1\",\"0\",\"0\",\"0\",\"19\",\"19\",\"1\",\"1\",\"2203\",\"0\",\"0.09641713\",\"1\",\"3\",\"0.023594523\",\"0.9894\",\"32.18\",\"1\",\"0.033029612\",\"0\",\"1.02078596947\",\"0.1103\",\"0.0523918\",\"0.1985\",\"1\",\"19\",\"1\",\"0.306909126508\",\"0\",\"0\",\"0\",\"1\",\"1\",\"0\",\"1\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"0\",\"32\",\"1\",\"3990\",\"0.0268534399436\",\"0.805897798425\",\"0.999749436232\",\"0.999749436232\",\"0.00399039504911\",\"0.99621077511\",\"23940\",\"21915511\",\"1.0119510009\",\"0.126509543309\",\"5438\",\"0\"],\"serving_default_input_app\":[\"93\",\"11.866646\",\"12.552132\",\"14.91172\",\"7.0242186\",\"-1.96152235015\",\"-1.36175023668\",\"-4.29921700914\",\"0.399202038495\",\"2.01828630522\",\"-1.13376638161\",\"0.999788135215\",\"1.74913257371\",\"-0.810501147957\",\"-2.37487956005\",\"2.16629469634\",\"-0.0908464407634\",\"4.08720124785\",\"-3.4242617182\",\"-3.34396755064\",\"-0.922276056989\",\"0.239808493591\",\"-0.397042896667\",\"-2.17235536906\",\"-4.5696074107\",\"-1.4068919915\",\"-2.09137522737\",\"-13.685564\",\"-12.191576\",\"-11.162037\",\"-13.695179\",\"119.72174761\",\"-399.82718185\",\"-259.414609226\",\"92.980296575\",\"-106.76884925\",\"54.330201587\",\"296.07348201\",\"201.46540676\",\"-207.18532521\",\"263.743178568\",\"-8.448718991\",\"-152.7029836\",\"175.51859297\",\"168.199568\",\"-13.0938081\",\"-85.7716733\",\"108.13419889\",\"-334.64929809\",\"251.194967712\",\"-3.449991865\",\"97.63811605\",\"-424.973489195\",\"-194.497896145\",\"0.083915666\",\"0.21310599\",\"-0.032019783\",\"-0.0027670665\",\"0.6127667\",\"0.54780114\",\"-0.9403887\",\"-0.13972458\",\"-0.23196886\",\"-0.33494487\",\"0.09588502\",\"0.1855776\",\"0.22639273\",\"-0.51628304\",\"-0.114166535\",\"-0.17727913\",\"0.16401836\",\"0.4561239\",\"-0.07739399\",\"-0.0013352555\",\"-0.11681123\",\"0.40487105\",\"-0.07625636\",\"-0.009285554\",\"0.29684278\",\"0.27495435\",\"0.08889192\",\"-0.21101737\",\"-0.09804925\",\"0.08848823\",\"0.35603377\",\"0.09004216\",\"0.017544597\",\"0.13777664\",\"0.19209054\",\"0.25220335\",\"-0.099241585\",\"0.13487557\",\"0.24160497\",\"0.15036109\",\"0.046898365\",\"0.18526399\",\"-0.119316176\",\"0.33652037\",\"-0.03923382\",\"0.048488438\",\"0.36614323\",\"0.23522355\",\"0.1696472\",\"0.07315683\",\"0.12800804\",\"0.7255761\",\"0.04787466\",\"0.26951617\",\"-0.022806883\",\"0.094774276\",\"-0.14585733\",\"0.40569603\",\"0.21212652\",\"-0.037195724\",\"0.102149636\",\"0.02303502\",\"-0.32382822\",\"-0.18476939\",\"-0.23270237\",\"-0.18768175\",\"0.34190777\",\"0.065103315\",\"-0.28565452\",\"-0.2506334\",\"-0.27315792\",\"0.8200944\",\"0.059797514\",\"0.16322254\",\"-0.2014275\",\"-0.16397618\",\"0.14908701\",\"0.042004257\",\"-0.21704954\",\"-0.31267768\",\"0.51314664\",\"-0.33213744\",\"0.31108487\",\"-0.3472758\",\"-0.031644046\",\"0.16807562\",\"0.16617405\",\"0.10363984\",\"0.07000345\",\"0.6260647\",\"0.60179526\",\"-9999\",\"0.59767187\",\"0.172441500884\",\"0.101480108686\",\"0.163144081831\",\"0.013918987\",\"0.004575706\",\"1\"],\"serving_default_input_pay\":[\"0.5\",\"0.1\",\"0\",\"0.1\",\"0\",\"0\",\"0\",\"0.1\",\"2\",\"1\",\"0.16666667\",\"5\",\"0\",\"0\",\"0\",\"0.5\",\"0\",\"0\",\"0\",\"0.5\",\"0.5\",\"0\",\"0\",\"2\",\"2\",\"0.18181819\",\"2\",\"0.45454547\",\"0.16666667\",\"1\",\"1\",\"1\",\"0\",\"0\",\"0\",\"0\",\"1\",\"0.5\",\"2\",\"1\",\"1\",\"715.72\",\"1\",\"1\",\"1754.52\",\"1\",\"1038.8\",\"0.0666666666667\",\"8224.85635359\",\"0.0165745856354\",\"0.0333333333333\",\"0.0295454545455\",\"0.7894737\",\"1\",\"0.44097406\",\"22\",\"0.7946768\",\"0.041430928\",\"934.4\",\"0.010600707\",\"3\",\"2\",\"2\",\"6\",\"8\",\"9\",\"1\",\"0.015675135\",\"44.7\",\"0\",\"4548.7\",\"0\",\"0.032967035\",\"3\",\"0\",\"0\",\"0.010416667\",\"0\",\"0\",\"1\",\"0.005952381\",\"3459.72\",\"3\",\"0.0215827338129\",\"6\",\"6\",\"1\",\"1\",\"1\",\"1\",\"1\",\"0.338444687842\",\"0.0111111111111\",\"0\",\"0.230769230769\",\"4\",\"10.21\",\"2\",\"304.88\",\"7\",\"11.79\",\"1\",\"0.128936170213\",\"0\",\"0.0348837209302\",\"3.99\",\"1\",\"10\",\"1\",\"10\",\"3.35061353862\",\"1.41232920053\",\"3.35061353862\",\"15\",\"0\",\"0.568657627683\",\"0\",\"1\",\"3\",\"0.023320526\",\"0.024752475\",\"0.9528302\",\"0.050552785\",\"0.016806724\",\"0.95652175\",\"0.84615386\",\"0\",\"0\",\"2\",\"0.034998618\",\"0.004201681\",\"8013.87\",\"0.00012056356\",\"2\"],\"serving_default_inner_pay_pay_type_seq\":[\"3\",\"3\",\"5\",\"5\",\"5\",\"5\",\"2\"],\"serving_default_input_graph\":[\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"0.0042347014\",\"NaN\",\"NaN\",\"NaN\",\"NaN\",\"0.0053616506\",\"NaN\",\"NaN\",\"0.016206454\",\"NaN\",\"NaN\"],\"serving_default_inner_pay_create_day_seq\":[\"2\",\"4\",\"4\",\"4\",\"4\",\"4\",\"5\"],\"serving_default_inner_pay_merchant_id_seq\":[\"1\",\"6\",\"1\",\"1\",\"3\",\"1\",\"1\"],\"serving_default_inner_pay_create_hour_seq\":[\"1\",\"4\",\"4\",\"1\",\"4\",\"5\",\"4\"],\"serving_default_input_bdsh\":[\"1.28571428571\",\"0.493068565006\",\"0.439294425001\",\"0.816111885592\",\"0\",\"0\",\"0\",\"0.001403309766532036\",\"5\",\"237260\",\"89270\",\"109840\",\"1\",\"0\",\"1\",\"1300\",\"0\",\"0.272727272727\",\"2\",\"0.881206430502\",\"0.273154075971\",\"0.657142857143\",\"0.98821473\",\"2.2652574\",\"1.4061607\",\"317641\",\"685659\"],\"serving_default_input_address\":[\"0.22043469381\",\"9014.395\",\"1.72085500352\",\"1.93596187896\",\"1.22448979592\",\"117.6\",\"97.4717094017\",\"11404.19\",\"1.26530612245\",\"300\",\"137\",\"1.75625\",\"1.96752533121\",\"-1\",\"0.129032258065\",\"0.548387096774\",\"2\",\"0.0645161290323\",\"0.156626506024\",\"0.156626506024\",\"0.614457831325\",\"0.0602409638554\",\"18\",\"0.148760330579\",\"17\",\"70\",\"0.578512396694\",\"8\",\"0.0661157024793\",\"30\",\"0.147783251232\",\"0.108374384236\",\"0.416666666667\",\"0\",\"0.291666666667\",\"0.0805418641406\",\"0.0839694656489\",\"0\",\"0\",\"0.0667425210323\",\"0\",\"1\",\"1\",\"0\",\"0\",\"6\",\"0.0227272727273\",\"1\",\"0\",\"0.0240749113167\",\"1\",\"1\",\"1\",\"0\",\"0\",\"2\"],\"serving_default_inner_pay_trade_amount_seq\":[\"3.4\",\"67.19\",\"5.9\",\"1.99\",\"69.9\",\"2.8\",\"1.9\"]}";
ObjectMapper mapper = new ObjectMapper();
TypeReference<HashMap<String, List<String>>> typeRef
= new TypeReference<>() {};
HashMap<String,List<String>> data = mapper.readValue(json, typeRef);
System.out.println("Got " + data);
// 100 1000 10000 100000 Please try it yourself
for (int i = 0; i < 100000; i++) {
var runner = kernel.runner();
var inputTensorList = new ArrayList<Tensor>();
data.forEach((var tensorName, var featureId) -> {
float[] seq = new float[featureId.size()];
for (int j = 0; j < featureId.size(); j++) {
seq[j] = Float.parseFloat(featureId.get(j));
}
var innerData = RawDataBufferFactory.create(seq, true);
var shape = Shape.of(1, seq.length);
var tensor = TFloat32.tensorOf(shape, innerData);
runner.feed(tensorName, tensor);
inputTensorList.add(tensor);
});
for (int j = 0; j < 2; j++) {
runner.fetch("StatefulPartitionedCall", j);
}
var output = runner.run();
var scores = new float[output.size()];
for (int j = 0; j < output.size(); j++) {
scores[j] = output.get(j).asRawTensor().data().asFloats().getFloat(0);
}
inputTensorList.forEach(Tensor::close);
output.close();
System.out.println(Arrays.toString(scores));
if (i % 1000==0) {
System.out.println("counter" + i);
}
}
}
public static Session loadTensorflowModel(String path) {
var config = ConfigProto.newBuilder()
.putDeviceCount("CPU", Runtime.getRuntime().availableProcessors())
.setInterOpParallelismThreads(8)
.setIntraOpParallelismThreads(8)
.setOperationTimeoutInMs(500)
.build();
var options = RunOptions.newBuilder()
.setTimeoutInMs(500)
.build();
var modelBundle = SavedModelBundle
.loader(path)
.withTags("serve")
.withConfigProto(config)
.withRunOptions(options)
.load();
return modelBundle.session();
}
} |
Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information
java -version
): java 8Describe the current behavior
we use "session.runner.run " to predict with tensors online,After hours of requests, the instance's memory kept increasing and eventually resulted in an out-of-memory (OOM) (or core dumped )error,
here is my behavior
At the start of the service, we load the session using the loadTensorflowModel method and persist it in memory. During runtime, a complete request process involves: 1. Feeding the Tensor using session.runner.feed, 2. Fetching the results using session.runner.fetch, 3. Obtaining the model's scoring results using the session.runner.run method, 4. Finally, closing both the input tensors and output tensors to complete the request and return the response.
Describe the expected behavior
I desire to keep the memory stable, maintaining it around 65%, rather than continuously increasing over time.
Code to reproduce the issue
Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
when i stop to request ,then the memory would gradually decrease until it returns to normal levels.
modelFile:after unzip, and load the path
test_model.zip
The text was updated successfully, but these errors were encountered: