Commit graph

109 commits

Author SHA1 Message Date
Victor Stinner 5dacbd4c42 Cleanup hashtable.h
Issue #26588:

* Pass the hash table rather than the key size to hash and compare functions
* _Py_HASHTABLE_READ_KEY() and _Py_HASHTABLE_ENTRY_READ_KEY() macros now expect
  the hash table as the first parameter, rather than the key size
* tracemalloc_get_traces_fill(): use _Py_HASHTABLE_ENTRY_READ_DATA() rather
  than pointer dereference
* Remove the _Py_HASHTABLE_ENTRY_WRITE_PKEY() macro
* Move "PKEY" and "PDATA" macros inside hashtable.c
2016-03-23 09:52:13 +01:00
Victor Stinner ca79ccd9e6 Issue #26588:
* Optimize tracemalloc_add_trace(): modify hashtable entry data (trace) if the
  memory block is already tracked, rather than trying to remove the old trace
  and then add a new trace.
* Add _Py_HASHTABLE_ENTRY_WRITE_DATA() macro
2016-03-23 09:38:54 +01:00
Victor Stinner e8c6b2fd1b Issue #26588:
* _Py_HASHTABLE_ENTRY_DATA: change type from "char *" to "const void *"
* Add _Py_HASHTABLE_ENTRY_WRITE_PKEY() macro
* Rename _Py_HASHTABLE_ENTRY_WRITE_DATA() macro to
  _Py_HASHTABLE_ENTRY_WRITE_PDATA()
* Add _Py_HASHTABLE_ENTRY_WRITE_DATA() macro
2016-03-23 09:25:01 +01:00
Victor Stinner 42bcf37fcf Issue #26588: Optimize tracemalloc_realloc()
No need to remove the old trace if the memory block didn't move.
2016-03-23 09:08:08 +01:00
Victor Stinner 35b0fd74dd Merge 3.5 2016-03-23 00:17:54 +01:00
Victor Stinner d5871e62dd Enhance _tracemalloc debug mode
Issue #26588: Enhance assertion in set_reentrant()
2016-03-23 00:17:04 +01:00
Victor Stinner 2f49e09110 Fix _tracemalloc start/stop
Issue #26588: Fix _tracemalloc start/stop: don't play with the reentrant flag.

set_reentrant(1) fails with an assertion error if tracemalloc_init() is called
first in a thread A and tracemalloc_start() is called second in a thread B. The
tracemalloc is imported in a thread A. Importing the module calls
tracemalloc_init(). tracemalloc.start() is called in a thread B.
2016-03-23 00:10:24 +01:00
Victor Stinner f9a71153e9 Issue #26588: remove debug traces from _tracemalloc. 2016-03-22 23:54:42 +01:00
Victor Stinner eddb074c53 Issue #26588: more debug traces 2016-03-22 21:06:07 +01:00
Victor Stinner b8d927266d Issue #26588: Don't call tracemalloc_init() at module initilization
So it's possible to get debug messages in test_tracemalloc.
2016-03-22 20:56:49 +01:00
Victor Stinner e31e35d34e Issue #26588: one more assertion 2016-03-22 18:48:50 +01:00
Victor Stinner 4a06647534 Add assertions on tracemalloc_reentrant_key
Issue #26588.
2016-03-22 17:45:09 +01:00
Victor Stinner 0cfc058d61 Issue #26588: more assertions 2016-03-22 17:40:07 +01:00
Victor Stinner 84aab09421 Issue #26588: add debug traces
Try to debug random failure on buildbots.
2016-03-22 16:13:31 +01:00
Victor Stinner 10b73e1748 Add C functions _PyTraceMalloc_Track()
Issue #26530:

* Add C functions _PyTraceMalloc_Track() and _PyTraceMalloc_Untrack() to track
  memory blocks using the tracemalloc module.
* Add _PyTraceMalloc_GetTraceback() to get the traceback of an object.
2016-03-22 13:39:05 +01:00
Victor Stinner e492ae50e2 tracemalloc now supports domains
Issue #26588:

* The _tracemalloc now supports tracing memory allocations of multiple address
  spaces (domains).
* Add domain parameter to tracemalloc_add_trace() and
  tracemalloc_remove_trace().
* tracemalloc_add_trace() now starts by removing the previous trace, if any.
* _tracemalloc._get_traces() now returns a list of (domain, size,
  traceback_frames): the domain is new.
* Add tracemalloc.DomainFilter
* tracemalloc.Filter: add an optional domain parameter to the constructor and a
  domain attribute
* Sublte change: use Py_uintptr_t rather than void* in the traces key.
* Add tracemalloc_config.use_domain, currently hardcoded to 1
2016-03-22 12:58:23 +01:00
Victor Stinner c9553876ae Simplify implementation of hashtable.c
Issue #26588: Remove copy_data, free_data and get_data_size callbacks from
hashtable.h. These callbacks are not used in Python and makes the code more
complex.

Remove also the _Py_HASHTABLE_ENTRY_DATA_AS_VOID_P() macro which uses an unsafe
pointer dereference (can cause memory alignment issue). Replace the macro usage
with _Py_HASHTABLE_ENTRY_READ_DATA() which is implemented with the safe
memcpy() function.
2016-03-22 12:13:01 +01:00
Victor Stinner b32a7edb22 Issue #26588: Fix compilation warning on Windows 2016-03-21 23:05:08 +01:00
Victor Stinner 285cf0a601 hashtable.h now supports keys of any size
Issue #26588: hashtable.h now supports keys of any size, not only
sizeof(void*). It allows to support key larger than sizeof(void*), but also to
use less memory for key smaller than sizeof(void*).
2016-03-21 22:00:58 +01:00
Victor Stinner 322bc12c31 Ooops, revert changeset ea9efa06c137
Change pushed by mistake, the patch is still under review :-/

"""
_tracemalloc: add domain to trace keys

* hashtable.h: key has now a variable size
* _tracemalloc uses (pointer: void*, domain: unsigned int) as key for traces
"""
2016-03-21 14:36:39 +01:00
Victor Stinner 51b846c47a _tracemalloc: add domain to trace keys
* hashtable.h: key has now a variable size
* _tracemalloc uses (pointer: void*, domain: unsigned int) as key for traces
2016-03-18 21:52:22 +01:00
Victor Stinner 0611c26a58 On memory error, dump the memory block traceback
Issue #26564: _PyObject_DebugDumpAddress() now dumps the traceback where a
memory block was allocated on memory block. Use the tracemalloc module to get
the traceback.
2016-03-15 22:22:13 +01:00
Victor Stinner af584a02a5 Merge 3.5 (_tracemalloc) 2016-03-15 21:57:23 +01:00
Victor Stinner 9528334e16 _tracemalloc: store lineno as unsigned int
Issue #26564. Cleanup the code, lineno is never negative.
2016-03-15 21:57:02 +01:00
Victor Stinner 7105e9f3de _tracemalloc: filename cannot be NULL 2016-03-15 14:28:04 +01:00
Victor Stinner d222653f8f Issue #26558: Remove useless check in tracemalloc
The first instruction of tracemalloc_add_trace() is traceback_new() which
already checks the GIL.
2016-03-14 17:01:32 +01:00
Victor Stinner cb7e5f6f08 Merge 3.4 (tracemalloc typo) 2015-03-18 16:05:32 +01:00
Victor Stinner 444f124fcf _tracemalloc.c: Fix typo 2015-03-18 16:05:18 +01:00
Serhiy Storchaka 56a6d855e2 Removed duplicated words in in comments and docs. 2014-12-01 18:28:43 +02:00
Serhiy Storchaka 83000a490a Removed duplicated words in in comments and docs. 2014-12-01 18:30:14 +02:00
Victor Stinner 52aee85613 Issue #22156: simplify _tracemalloc.c, use an int for the MAX_NFRAME constant 2014-08-16 15:44:02 +02:00
Victor Stinner 706768c687 Issue #22156: Fix some "comparison between signed and unsigned integers"
compiler warnings in the Modules/ subdirectory.
2014-08-16 01:03:39 +02:00
Victor Stinner d8f0d922d5 Issue #21233: Rename the C structure "PyMemAllocator" to "PyMemAllocatorEx" to
make sure that the code using it will be adapted for the new "calloc" field
(instead of crashing).
2014-06-02 21:57:10 +02:00
Victor Stinner aa0e7afa43 Issue #21639: Fix a division by zero in tracemalloc on calloc(0, 0). The
regression was introduced recently with the introduction of the new "calloc"
functions (PyMem_RawCalloc, PyMem_Calloc, PyObject_Calloc).

Add also a unit test to check for the non-regression.
2014-06-02 21:40:22 +02:00
Victor Stinner 45e8e2f218 Issue #21490: Add new C macros: Py_ABS() and Py_STRINGIFY()
Keep _Py_STRINGIZE() in PC/pyconfig.h to not introduce a dependency between
pyconfig.h and pymacros.h.
2014-05-14 17:24:35 +02:00
Victor Stinner db067af12a Issue #21233: Add new C functions: PyMem_RawCalloc(), PyMem_Calloc(),
PyObject_Calloc(), _PyObject_GC_Calloc(). bytes(int) and bytearray(int) are now
using ``calloc()`` instead of ``malloc()`` for large objects which is faster
and use less memory (until the bytearray buffer is filled with data).
2014-05-02 22:31:14 +02:00
Victor Stinner d9a7352348 Issue #21036: Fix typo in macro name
_PY_HASHTABLE_ENTRY_DATA => _Py_HASHTABLE_ENTRY_DATA
2014-03-24 22:34:34 +01:00
Victor Stinner dd382ef8ec Issue #20354: Fix alignment issue in the tracemalloc module on 64-bit
platforms. Bug seen on 64-bit Linux when using "make profile-opt".

Only align the "frame_t" structure on 32-bit when Visual Studio is used. Before
the alignment to 32-bit was applied to the whole file any compiler supporting
"#pragma pack(4)" which includes GCC.
2014-02-01 03:43:58 +01:00
Victor Stinner 711e27cd50 tracemalloc: fix usage of strtol(), value can be LONG_MIN or LONG_MAX on ERANGE 2013-12-16 23:06:19 +01:00
Victor Stinner 4d8c29cd57 tracemalloc: only use unsigned types to compute hash
Commit to simplify the backport to python 2.7 and to make the code more
consistent.
2013-12-16 23:05:13 +01:00
Victor Stinner 590cebe391 Issue #19787: PyThread_set_key_value() now always set the value
In Python 3.3, PyThread_set_key_value() did nothing if the key already exists
(if the current value is a non-NULL pointer).

When _PyGILState_NoteThreadState() is called twice on the same thread with a
different Python thread state, it still keeps the old Python thread state to
keep the old behaviour. Replacing the Python thread state with the new state
introduces new bugs: see issues #10915 and #15751.
2013-12-13 11:08:56 +01:00
Victor Stinner 441adb8c57 Backout changeset 46393019b650
test_capi is failing and the fix is not trivial, I prefer to revert
2013-12-13 04:14:41 +01:00
Victor Stinner cd0cb8ccd3 Close #19787: PyThread_set_key_value() now always set the value. In Python 3.3,
the function did nothing if the key already exists (if the current value is a
non-NULL pointer).

_testcapi.run_in_subinterp() now correctly sets the new Python thread state of
the current thread when a subinterpreter is created.
2013-12-13 03:22:00 +01:00
Victor Stinner 88c29877c7 Close #19741: tracemalloc_realloc() does not release the table lock anymore
between tracemalloc_remove_trace() and tracemalloc_add_trace() to reduce the
risk of race condition.

tracemalloc_add_trace() cannot fail anymore in tracemalloc_realloc() when
tracemalloc_realloc() resizes a memory block.
2013-12-04 01:47:46 +01:00
Victor Stinner 1511680b79 Close #19757: Cleanup tracemalloc, move
PyGILState_Ensure()/PyGILState_Release() calls to the raw wrappers to simplify
the code.

Rename also tracemalloc_log_alloc/log_free() to
tracemalloc_add_trace/remove_trace().
2013-12-04 01:29:35 +01:00
Victor Stinner be0708f066 Closes #19831: Stop tracemalloc later at Python shutdown to be able to use
tracemalloc in objects destructor

Replace atexit handler with an harcoded C function _PyTraceMalloc_Fini().
2013-12-01 10:03:26 +01:00
Victor Stinner f28ce60441 Closes #19786: tracemalloc, remove the arbitrary limit of 100 frames
The limit is now 178,956,969 on 64 bit (it is greater on 32 bit because
structures are smaller).

Use int instead of Py_ssize_t to store the number of frames to have smaller
traceback_t objects.
2013-11-27 22:27:13 +01:00
Victor Stinner 3c0481d426 Close #19798: replace "maximum" term with "peak" in get_traced_memory()
documentation. Use also the term "current" for the current size.
2013-11-27 21:39:49 +01:00
Victor Stinner 59463d8340 tracemalloc: fix get_traced_memory() docstring for result type 2013-11-26 10:46:06 +01:00
Victor Stinner 4dc74204c1 Issue #18874: Fix typo 2013-11-26 01:18:52 +01:00
Victor Stinner 2ead3d2448 Issue #18874: make it more explicit than set_reentrant() only accept 0 or 1 2013-11-26 01:08:53 +01:00
Victor Stinner 7a5be14dd8 Issue #18874: tracemalloc: Comment the trace_t structure 2013-11-26 01:06:02 +01:00
Victor Stinner de2f132a5e Issue #18874: tracemalloc: explain the purpose of get_traces.tracebacks in a comment 2013-11-26 00:26:23 +01:00
Victor Stinner 4fbefdb217 Close #19762: Fix name of _get_traces() and _get_object_traceback() function
name in their docstring. Patch written by Vajrasky Kok.
2013-11-25 09:33:18 +01:00
Victor Stinner 08facd2009 Issue #19741: cleanup tracemalloc_realloc()
Explain that unhandled error case is very unlikely
2013-11-24 12:27:59 +01:00
Victor Stinner 52968676f1 Issue #19741: tracemalloc: report tracemalloc_log_alloc() failure to the caller
for new allocations, but not when a memory block was already resized
2013-11-24 11:37:15 +01:00
Victor Stinner d606ba7f55 Issue #19741: fix tracemalloc_log_alloc(), handle _Py_HASHTABLE_SET() failure 2013-11-24 11:28:20 +01:00
Victor Stinner 3728d6ced0 Issue #18874: Remove tracemalloc.set_traceback_limit()
tracemalloc.start() now has an option nframe parameter
2013-11-23 12:37:20 +01:00
Victor Stinner ed3b0bca3e Issue #18874: Implement the PEP 454 (tracemalloc) 2013-11-23 12:27:24 +01:00