Skip to content

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
    • Help
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
W
wendelin.core
  • Project
    • Project
    • Details
    • Activity
    • Releases
    • Cycle Analytics
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Charts
  • Issues 1
    • Issues 1
    • List
    • Board
    • Labels
    • Milestones
  • Merge Requests 0
    • Merge Requests 0
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
    • Charts
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Charts
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • nexedi
  • wendelin.core
  • Issues
  • #3

Closed
Open
Opened Jul 21, 2016 by Kirill Smelkov@kirr
  • Report abuse
  • New issue
Report abuse New issue

NEO / w.c test -> RAM leak

(reported by @Tyagov)

Hi, I let it run over night my quick feed into NEO w.c. array.

Now simply out of space and array shape is

(23379050506,)

Here's my script:

This script will consume as much memory as machine has an eventually it will be killed by kernel. So it's worth investigating why we have this leak and why garbage collection not working with w.c. array (my wild guess now)

from wendelin.bigarray.array_zodb import ZBigArray
from wendelin.lib.zodb import dbopen, dbclose
import transaction
import numpy as np

root = dbopen('neo://neo-standalone-comp-2477@[2001:67c:1254:YY::ZZZZ]:2051')
#root['B'] = A = ZBigArray((10,), np.float64)
#transaction.commit()


A = root['B']

slice_index = 10000
shape = A.shape[0]

print shape

# iterate over array
for i in xrange(0, shape, slice_index):
  np.average(A[i:i+slice_index])


#for i in range(20000):
#  A.append( np.random.rand(4*1024*1024)   ) #np.arange(4*1024*1024) )
#  print i, A.shape
#  transaction.commit()

To simulate:

  1. ssh root@2001:67c:1254:XX::1

  2. su ivan; cd /home/ivan

  3. . test/bin/activate

  4. ipython feed.py

Regards Ivan

Assignee
Assign to
None
Milestone
None
Assign milestone
Time tracking
None
Due date
None
0
Labels
None
Assign labels
  • View project labels
Reference: nexedi/wendelin.core#3
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7