C# “分块”内存流
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/1203121/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
提示：将鼠标放在中文语句上可以显示对应的英文。显示中英文
时间：2020-08-06 10:48:36 来源:igfitidea点击:
"Chunked" MemoryStream
提问by Karol Kolenda
I'm looking for the implementation of MemoryStream which does not allocate memory as one big block, but rather a collection of chunks. I want to store a few GB of data in memory (64 bit) and avoid limitation of memory fragmentation.
我正在寻找 MemoryStream 的实现，它不会将内存分配为一个大块，而是一组块。我想在内存（64 位）中存储几 GB 的数据并避免内存碎片的限制。
采纳答案by chuckj
You need to first determine if virtual address fragmentation is the problem.
您需要首先确定虚拟地址碎片是否是问题所在。
If you are on a 64 bit machine (which you seem to indicate you are) I seriously doubt it is. Each 64 bit process has almost the the entire 64 bit virtual memory space available and your only worry is virtual address space fragmentation not physical memory fragmentation (which is what the operating system must worry about). The OS memory manager already pages memory under the covers. For the forseeable future you will not run out of virtual address space before you run out of physical memory. This is unlikely change before we both retire.
如果您使用的是 64 位机器（您似乎表明您是），我严重怀疑它是。每个 64 位进程几乎都有整个 64 位虚拟内存空间可用，您唯一担心的是虚拟地址空间碎片而不是物理内存碎片（这是操作系统必须担心的）。操作系统内存管理器已经在幕后分页内存。在可预见的未来，在耗尽物理内存之前，您将不会耗尽虚拟地址空间。在我们都退休之前，这不太可能发生变化。
If you are have a 32 bit address space, then allocating contiguous large blocks of memory in the GB ramge you will encounter a fragmentation problem quite quickly. There is no stock chunk allocating memory stream in the CLR. There is one in the under the covers in ASP.NET (for other reasons) but it is not accessable. If you must travel this path you are probably better off writing one youself anyway because the usage pattern of your application is unlikely to be similar to many others and trying to fit your data into a 32bit address space will likely be your perf bottleneck.
如果您有一个 32 位地址空间，那么在 GB ramge 中分配连续的大内存块时，您将很快遇到碎片问题。CLR 中没有用于分配内存流的库存块。在 ASP.NET 的幕后有一个（出于其他原因），但无法访问。如果您必须走这条路，那么您最好还是自己编写一个，因为您的应用程序的使用模式不太可能与许多其他应用程序相似，并且尝试将您的数据放入 32 位地址空间可能会成为您的性能瓶颈。
I highly recommend requiring a 64 bit process if you are manipulating GBs of data. It will do a much better job than hand-rolled solutions to 32 bit address space fragmentation regardless of how cleaver you are.
如果您要操作 GB 的数据，我强烈建议您使用 64 位进程。无论您有多切割，它都会比针对 32 位地址空间碎片的手动解决方案做得更好。
回答by Daniel Earwicker
Something like this:
像这样的东西：
class ChunkedMemoryStream : Stream
{
private readonly List<byte[]> _chunks = new List<byte[]>();
private int _positionChunk;
private int _positionOffset;
private long _position;
public override bool CanRead
{
get { return true; }
}
public override bool CanSeek
{
get { return true; }
}
public override bool CanWrite
{
get { return true; }
}
public override void Flush() { }
public override long Length
{
get { return _chunks.Sum(c => c.Length); }
}
public override long Position
{
get
{
return _position;
}
set
{
_position = value;
_positionChunk = 0;
while (_positionOffset != 0)
{
if (_positionChunk >= _chunks.Count)
throw new OverflowException();
if (_positionOffset < _chunks[_positionChunk].Length)
return;
_positionOffset -= _chunks[_positionChunk].Length;
_positionChunk++;
}
}
}
public override int Read(byte[] buffer, int offset, int count)
{
int result = 0;
while ((count != 0) && (_positionChunk != _chunks.Count))
{
int fromChunk = Math.Min(count, _chunks[_positionChunk].Length - _positionOffset);
if (fromChunk != 0)
{
Array.Copy(_chunks[_positionChunk], _positionOffset, buffer, offset, fromChunk);
offset += fromChunk;
count -= fromChunk;
result += fromChunk;
_position += fromChunk;
}
_positionOffset = 0;
_positionChunk++;
}
return result;
}
public override long Seek(long offset, SeekOrigin origin)
{
long newPos = 0;
switch (origin)
{
case SeekOrigin.Begin:
newPos = offset;
break;
case SeekOrigin.Current:
newPos = Position + offset;
break;
case SeekOrigin.End:
newPos = Length - offset;
break;
}
Position = Math.Max(0, Math.Min(newPos, Length));
return newPos;
}
public override void SetLength(long value)
{
throw new NotImplementedException();
}
public override void Write(byte[] buffer, int offset, int count)
{
while ((count != 0) && (_positionChunk != _chunks.Count))
{
int toChunk = Math.Min(count, _chunks[_positionChunk].Length - _positionOffset);
if (toChunk != 0)
{
Array.Copy(buffer, offset, _chunks[_positionChunk], _positionOffset, toChunk);
offset += toChunk;
count -= toChunk;
_position += toChunk;
}
_positionOffset = 0;
_positionChunk++;
}
if (count != 0)
{
byte[] chunk = new byte[count];
Array.Copy(buffer, offset, chunk, 0, count);
_chunks.Add(chunk);
_positionChunk = _chunks.Count;
_position += count;
}
}
}
class Program
{
static void Main(string[] args)
{
ChunkedMemoryStream cms = new ChunkedMemoryStream();
Debug.Assert(cms.Length == 0);
Debug.Assert(cms.Position == 0);
cms.Position = 0;
byte[] helloworld = Encoding.UTF8.GetBytes("hello world");
cms.Write(helloworld, 0, 3);
cms.Write(helloworld, 3, 3);
cms.Write(helloworld, 6, 5);
Debug.Assert(cms.Length == 11);
Debug.Assert(cms.Position == 11);
cms.Position = 0;
byte[] b = new byte[20];
cms.Read(b, 3, (int)cms.Length);
Debug.Assert(b.Skip(3).Take(11).SequenceEqual(helloworld));
cms.Position = 0;
cms.Write(Encoding.UTF8.GetBytes("seeya"), 0, 5);
Debug.Assert(cms.Length == 11);
Debug.Assert(cms.Position == 5);
cms.Position = 0;
cms.Read(b, 0, (byte) cms.Length);
Debug.Assert(b.Take(11).SequenceEqual(Encoding.UTF8.GetBytes("seeya world")));
Debug.Assert(cms.Length == 11);
Debug.Assert(cms.Position == 11);
cms.Write(Encoding.UTF8.GetBytes(" again"), 0, 6);
Debug.Assert(cms.Length == 17);
Debug.Assert(cms.Position == 17);
cms.Position = 0;
cms.Read(b, 0, (byte)cms.Length);
Debug.Assert(b.Take(17).SequenceEqual(Encoding.UTF8.GetBytes("seeya world again")));
}
}
回答by user293217
回答by Rudolf Dvoracek
I've found similar problem in my application. I've read large amount of compressed data and I suffered from OutOfMemoryException using MemoryStream. I've written my own implementation of "chunked" memory stream based on collection of byte arrays. If you have any idea how to make this memory stream more effective, please write me about it.
我在我的应用程序中发现了类似的问题。我已经读取了大量压缩数据，并且使用 MemoryStream 遇到了 OutOfMemoryException。我已经根据字节数组的集合编写了自己的“分块”内存流实现。如果您有任何想法如何使此内存流更有效，请写信给我。
public sealed class ChunkedMemoryStream : Stream
{
#region Constants
private const int BUFFER_LENGTH = 65536;
private const byte ONE = 1;
private const byte ZERO = 0;
#endregion
#region Readonly & Static Fields
private readonly Collection<byte[]> _chunks;
#endregion
#region Fields
private long _length;
private long _position;
private const byte TWO = 2;
#endregion
#region C'tors
public ChunkedMemoryStream()
{
_chunks = new Collection<byte[]> { new byte[BUFFER_LENGTH], new byte[BUFFER_LENGTH] };
_position = ZERO;
_length = ZERO;
}
#endregion
#region Instance Properties
public override bool CanRead
{
get { return true; }
}
public override bool CanSeek
{
get { return true; }
}
public override bool CanWrite
{
get { return true; }
}
public override long Length
{
get { return _length; }
}
public override long Position
{
get { return _position; }
set
{
if (!CanSeek)
throw new NotSupportedException();
_position = value;
if (_position > _length)
_position = _length - ONE;
}
}
private byte[] CurrentChunk
{
get
{
long positionDividedByBufferLength = _position / BUFFER_LENGTH;
var chunkIndex = Convert.ToInt32(positionDividedByBufferLength);
byte[] chunk = _chunks[chunkIndex];
return chunk;
}
}
private int PositionInChunk
{
get
{
int positionInChunk = Convert.ToInt32(_position % BUFFER_LENGTH);
return positionInChunk;
}
}
private int RemainingBytesInCurrentChunk
{
get
{
Contract.Ensures(Contract.Result<int>() > ZERO);
int remainingBytesInCurrentChunk = CurrentChunk.Length - PositionInChunk;
return remainingBytesInCurrentChunk;
}
}
#endregion
#region Instance Methods
public override void Flush()
{
}
public override int Read(byte[] buffer, int offset, int count)
{
if (offset + count > buffer.Length)
throw new ArgumentException();
if (buffer == null)
throw new ArgumentNullException();
if (offset < ZERO || count < ZERO)
throw new ArgumentOutOfRangeException();
if (!CanRead)
throw new NotSupportedException();
int bytesToRead = count;
if (_length - _position < bytesToRead)
bytesToRead = Convert.ToInt32(_length - _position);
int bytesreaded = 0;
while (bytesToRead > ZERO)
{
// get remaining bytes in current chunk
// read bytes in current chunk
// advance to next position
int remainingBytesInCurrentChunk = RemainingBytesInCurrentChunk;
if (remainingBytesInCurrentChunk > bytesToRead)
remainingBytesInCurrentChunk = bytesToRead;
Array.Copy(CurrentChunk, PositionInChunk, buffer, offset, remainingBytesInCurrentChunk);
//move position in source
_position += remainingBytesInCurrentChunk;
//move position in target
offset += remainingBytesInCurrentChunk;
//bytesToRead is smaller
bytesToRead -= remainingBytesInCurrentChunk;
//count readed bytes;
bytesreaded += remainingBytesInCurrentChunk;
}
return bytesreaded;
}
public override long Seek(long offset, SeekOrigin origin)
{
switch (origin)
{
case SeekOrigin.Begin:
Position = offset;
break;
case SeekOrigin.Current:
Position += offset;
break;
case SeekOrigin.End:
Position = Length + offset;
break;
}
return Position;
}
private long Capacity
{
get
{
int numberOfChunks = _chunks.Count;
long capacity = numberOfChunks * BUFFER_LENGTH;
return capacity;
}
}
public override void SetLength(long value)
{
if (value > _length)
{
while (value > Capacity)
{
var item = new byte[BUFFER_LENGTH];
_chunks.Add(item);
}
}
else if (value < _length)
{
var decimalValue = Convert.ToDecimal(value);
var valueToBeCompared = decimalValue % BUFFER_LENGTH == ZERO ? Capacity : Capacity - BUFFER_LENGTH;
//remove data chunks, but leave at least two chunks
while (value < valueToBeCompared && _chunks.Count > TWO)
{
byte[] lastChunk = _chunks.Last();
_chunks.Remove(lastChunk);
}
}
_length = value;
if (_position > _length - ONE)
_position = _length == 0 ? ZERO : _length - ONE;
}
public override void Write(byte[] buffer, int offset, int count)
{
if (!CanWrite)
throw new NotSupportedException();
int bytesToWrite = count;
while (bytesToWrite > ZERO)
{
//get remaining space in current chunk
int remainingBytesInCurrentChunk = RemainingBytesInCurrentChunk;
//if count of bytes to be written is fewer than remaining
if (remainingBytesInCurrentChunk > bytesToWrite)
remainingBytesInCurrentChunk = bytesToWrite;
//if remaining bytes is still greater than zero
if (remainingBytesInCurrentChunk > ZERO)
{
//write remaining bytes to current Chunk
Array.Copy(buffer, offset, CurrentChunk, PositionInChunk, remainingBytesInCurrentChunk);
//change offset of source array
offset += remainingBytesInCurrentChunk;
//change bytes to write
bytesToWrite -= remainingBytesInCurrentChunk;
//change length and position
_length += remainingBytesInCurrentChunk;
_position += remainingBytesInCurrentChunk;
}
if (Capacity == _position)
_chunks.Add(new byte[BUFFER_LENGTH]);
}
}
/// <summary>
/// Gets entire content of stream regardless of Position value and return output as byte array
/// </summary>
/// <returns>byte array</returns>
public byte[] ToArray()
{
var outputArray = new byte[Length];
if (outputArray.Length != ZERO)
{
long outputPosition = ZERO;
foreach (byte[] chunk in _chunks)
{
var remainingLength = (Length - outputPosition) > chunk.Length
? chunk.Length
: Length - outputPosition;
Array.Copy(chunk, ZERO, outputArray, outputPosition, remainingLength);
outputPosition = outputPosition + remainingLength;
}
}
return outputArray;
}
/// <summary>
/// Method set Position to first element and write entire stream to another
/// </summary>
/// <param name="stream">Target stream</param>
public void WriteTo(Stream stream)
{
Contract.Requires(stream != null);
Position = ZERO;
var buffer = new byte[BUFFER_LENGTH];
int bytesReaded;
do
{
bytesReaded = Read(buffer, ZERO, BUFFER_LENGTH);
stream.Write(buffer, ZERO, bytesReaded);
} while (bytesReaded > ZERO);
}
#endregion
}
回答by Richard Anthony Hein
You should use the UnmanagedMemoryStreamwhen dealing with over 2GB chunks of memory, as MemoryStream is limited to 2GB, and the UnmanagedMemoryStreamwas made to deal with this problem.
在处理超过 2GB 的内存块时，您应该使用UnmanagedMemoryStream，因为 MemoryStream 被限制为 2GB，而UnmanagedMemoryStream就是用来处理这个问题的。
回答by BrainSlugs83
SparseMemoryStream does this in .NET it's in buried deep down in an internal class library though -- the source code is available of course, since Microsoft put it all out there as open source.
SparseMemoryStream 在 .NET 中做到了这一点，尽管它深埋在内部类库中——源代码当然是可用的，因为微软把它全部作为开源发布。
You can grab the code for it here: http://www.dotnetframework.org/default.aspx/4@0/4@0/DEVDIV_TFS/Dev10/Releases/RTMRel/wpf/src/Base/MS/Internal/IO/Packaging/SparseMemoryStream@cs/1305600/SparseMemoryStream@cs
你可以在这里获取它的代码：http: //www.dotnetframework.org/default.aspx/4@0/4@0/DEVDIV_TFS/Dev10/Releases/RTMRel/wpf/src/Base/MS/Internal/IO /Packaging/SparseMemoryStream@cs/1305600/SparseMemoryStream@cs
That being said, I highly recommend not using it as is -- At the very least remove all the calls to IsolatedStorage for starters, as this seems to be the cause of no end of bugs* in the framework's packaging API.
话虽如此，我强烈建议不要按原样使用它——至少为初学者删除所有对 IndependentStorage 的调用，因为这似乎是框架打包 API 中 bug 无穷无尽的原因。
(*: In addition to spreading the data around in streams, if it gets too big, it basically reinvents swap files for some reason -- in the user's Isolated Storage no less -- and coincidentally, most MS products that allow for .NET based add-ins do not have their app domains setup in such a way that you can access Isolated Storage -- VSTO add-ins are notorious for suffering from this issue, for example.)
(*: 除了在流中传播数据之外，如果它变得太大，它基本上会出于某种原因重新发明交换文件——在用户的隔离存储中不少——巧合的是，大多数允许基于 .NET 的 MS 产品加载项没有以您可以访问独立存储的方式设置其应用程序域——例如，VSTO 加载项因遭受此问题而臭名昭著。）
回答by Simon Mourier
Here is a full implementation:
这是一个完整的实现：
/// <summary>
/// Defines a MemoryStream that does not sit on the Large Object Heap, thus avoiding memory fragmentation.
/// </summary>
public sealed class ChunkedMemoryStream : Stream
{
/// <summary>
/// Defines the default chunk size. Currently defined as 0x10000.
/// </summary>
public const int DefaultChunkSize = 0x10000; // needs to be < 85000
private List<byte[]> _chunks = new List<byte[]>();
private long _position;
private int _chunkSize;
private int _lastChunkPos;
private int _lastChunkPosIndex;
/// <summary>
/// Initializes a new instance of the <see cref="ChunkedMemoryStream"/> class.
/// </summary>
public ChunkedMemoryStream()
: this(DefaultChunkSize)
{
}
/// <summary>
/// Initializes a new instance of the <see cref="ChunkedMemoryStream"/> class.
/// </summary>
/// <param name="chunkSize">Size of the underlying chunks.</param>
public ChunkedMemoryStream(int chunkSize)
: this(null)
{
}
/// <summary>
/// Initializes a new instance of the <see cref="ChunkedMemoryStream"/> class based on the specified byte array.
/// </summary>
/// <param name="buffer">The array of unsigned bytes from which to create the current stream.</param>
public ChunkedMemoryStream(byte[] buffer)
: this(DefaultChunkSize, buffer)
{
}
/// <summary>
/// Initializes a new instance of the <see cref="ChunkedMemoryStream"/> class based on the specified byte array.
/// </summary>
/// <param name="chunkSize">Size of the underlying chunks.</param>
/// <param name="buffer">The array of unsigned bytes from which to create the current stream.</param>
public ChunkedMemoryStream(int chunkSize, byte[] buffer)
{
FreeOnDispose = true;
ChunkSize = chunkSize;
_chunks.Add(new byte[chunkSize]);
if (buffer != null)
{
Write(buffer, 0, buffer.Length);
Position = 0;
}
}
/// <summary>
/// Gets or sets a value indicating whether to free the underlying chunks on dispose.
/// </summary>
/// <value><c>true</c> if [free on dispose]; otherwise, <c>false</c>.</value>
public bool FreeOnDispose { get; set; }
/// <summary>
/// Releases the unmanaged resources used by the <see cref="T:System.IO.Stream"/> and optionally releases the managed resources.
/// </summary>
/// <param name="disposing">true to release both managed and unmanaged resources; false to release only unmanaged resources.</param>
protected override void Dispose(bool disposing)
{
if (FreeOnDispose)
{
if (_chunks != null)
{
_chunks = null;
_chunkSize = 0;
_position = 0;
}
}
base.Dispose(disposing);
}
/// <summary>
/// When overridden in a derived class, clears all buffers for this stream and causes any buffered data to be written to the underlying device.
/// This implementation does nothing.
/// </summary>
public override void Flush()
{
// do nothing
}
/// <summary>
/// When overridden in a derived class, reads a sequence of bytes from the current stream and advances the position within the stream by the number of bytes read.
/// </summary>
/// <param name="buffer">An array of bytes. When this method returns, the buffer contains the specified byte array with the values between <paramref name="offset"/> and (<paramref name="offset"/> + <paramref name="count"/> - 1) replaced by the bytes read from the current source.</param>
/// <param name="offset">The zero-based byte offset in <paramref name="buffer"/> at which to begin storing the data read from the current stream.</param>
/// <param name="count">The maximum number of bytes to be read from the current stream.</param>
/// <returns>
/// The total number of bytes read into the buffer. This can be less than the number of bytes requested if that many bytes are not currently available, or zero (0) if the end of the stream has been reached.
/// </returns>
/// <exception cref="T:System.ArgumentException">
/// The sum of <paramref name="offset"/> and <paramref name="count"/> is larger than the buffer length.
/// </exception>
/// <exception cref="T:System.ArgumentNullException">
/// <paramref name="buffer"/> is null.
/// </exception>
/// <exception cref="T:System.ArgumentOutOfRangeException">
/// <paramref name="offset"/> or <paramref name="count"/> is negative.
/// </exception>
/// <exception cref="T:System.ObjectDisposedException">
/// Methods were called after the stream was closed.
/// </exception>
public override int Read(byte[] buffer, int offset, int count)
{
if (buffer == null)
throw new ArgumentNullException("buffer");
if (offset < 0)
throw new ArgumentOutOfRangeException("offset");
if (count < 0)
throw new ArgumentOutOfRangeException("count");
if ((buffer.Length - offset) < count)
throw new ArgumentException(null, "count");
CheckDisposed();
int chunkIndex = (int)(_position / ChunkSize);
if (chunkIndex == _chunks.Count)
return 0;
int chunkPos = (int)(_position % ChunkSize);
count = (int)Math.Min(count, Length - _position);
if (count == 0)
return 0;
int left = count;
int inOffset = offset;
int total = 0;
do
{
int toCopy = Math.Min(left, ChunkSize - chunkPos);
Buffer.BlockCopy(_chunks[chunkIndex], chunkPos, buffer, inOffset, toCopy);
inOffset += toCopy;
left -= toCopy;
total += toCopy;
if ((chunkPos + toCopy) == ChunkSize)
{
if (chunkIndex == (_chunks.Count - 1))
{
// last chunk
break;
}
chunkPos = 0;
chunkIndex++;
}
else
{
chunkPos += toCopy;
}
}
while (left > 0);
_position += total;
return total;
}
/// <summary>
/// Reads a byte from the stream and advances the position within the stream by one byte, or returns -1 if at the end of the stream.
/// </summary>
/// <returns>
/// The unsigned byte cast to an Int32, or -1 if at the end of the stream.
/// </returns>
/// <exception cref="T:System.ObjectDisposedException">
/// Methods were called after the stream was closed.
/// </exception>
public override int ReadByte()
{
CheckDisposed();
if (_position >= Length)
return -1;
byte b = _chunks[(int)(_position / ChunkSize)][_position % ChunkSize];
_position++;
return b;
}
/// <summary>
/// When overridden in a derived class, sets the position within the current stream.
/// </summary>
/// <param name="offset">A byte offset relative to the <paramref name="origin"/> parameter.</param>
/// <param name="origin">A value of type <see cref="T:System.IO.SeekOrigin"/> indicating the reference point used to obtain the new position.</param>
/// <returns>
/// The new position within the current stream.
/// </returns>
/// <exception cref="T:System.ObjectDisposedException">
/// Methods were called after the stream was closed.
/// </exception>
public override long Seek(long offset, SeekOrigin origin)
{
CheckDisposed();
switch (origin)
{
case SeekOrigin.Begin:
Position = offset;
break;
case SeekOrigin.Current:
Position += offset;
break;
case SeekOrigin.End:
Position = Length + offset;
break;
}
return Position;
}
private void CheckDisposed()
{
if (_chunks == null)
throw new ObjectDisposedException(null, "Cannot access a disposed stream");
}
/// <summary>
/// When overridden in a derived class, sets the length of the current stream.
/// </summary>
/// <param name="value">The desired length of the current stream in bytes.</param>
/// <exception cref="T:System.ObjectDisposedException">
/// Methods were called after the stream was closed.
/// </exception>
public override void SetLength(long value)
{
CheckDisposed();
if (value < 0)
throw new ArgumentOutOfRangeException("value");
if (value > Length)
throw new ArgumentOutOfRangeException("value");
long needed = value / ChunkSize;
if ((value % ChunkSize) != 0)
{
needed++;
}
if (needed > int.MaxValue)
throw new ArgumentOutOfRangeException("value");
if (needed < _chunks.Count)
{
int remove = (int)(_chunks.Count - needed);
for (int i = 0; i < remove; i++)
{
_chunks.RemoveAt(_chunks.Count - 1);
}
}
_lastChunkPos = (int)(value % ChunkSize);
}
/// <summary>
/// Converts the current stream to a byte array.
/// </summary>
/// <returns>An array of bytes</returns>
public byte[] ToArray()
{
CheckDisposed();
byte[] bytes = new byte[Length];
int offset = 0;
for (int i = 0; i < _chunks.Count; i++)
{
int count = (i == (_chunks.Count - 1)) ? _lastChunkPos : _chunks[i].Length;
if (count > 0)
{
Buffer.BlockCopy(_chunks[i], 0, bytes, offset, count);
offset += count;
}
}
return bytes;
}
/// <summary>
/// When overridden in a derived class, writes a sequence of bytes to the current stream and advances the current position within this stream by the number of bytes written.
/// </summary>
/// <param name="buffer">An array of bytes. This method copies <paramref name="count"/> bytes from <paramref name="buffer"/> to the current stream.</param>
/// <param name="offset">The zero-based byte offset in <paramref name="buffer"/> at which to begin copying bytes to the current stream.</param>
/// <param name="count">The number of bytes to be written to the current stream.</param>
/// <exception cref="T:System.ArgumentException">
/// The sum of <paramref name="offset"/> and <paramref name="count"/> is greater than the buffer length.
/// </exception>
/// <exception cref="T:System.ArgumentNullException">
/// <paramref name="buffer"/> is null.
/// </exception>
/// <exception cref="T:System.ArgumentOutOfRangeException">
/// <paramref name="offset"/> or <paramref name="count"/> is negative.
/// </exception>
/// <exception cref="T:System.ObjectDisposedException">
/// Methods were called after the stream was closed.
/// </exception>
public override void Write(byte[] buffer, int offset, int count)
{
if (buffer == null)
throw new ArgumentNullException("buffer");
if (offset < 0)
throw new ArgumentOutOfRangeException("offset");
if (count < 0)
throw new ArgumentOutOfRangeException("count");
if ((buffer.Length - offset) < count)
throw new ArgumentException(null, "count");
CheckDisposed();
int chunkPos = (int)(_position % ChunkSize);
int chunkIndex = (int)(_position / ChunkSize);
if (chunkIndex == _chunks.Count)
{
_chunks.Add(new byte[ChunkSize]);
}
int left = count;
int inOffset = offset;
do
{
int copied = Math.Min(left, ChunkSize - chunkPos);
Buffer.BlockCopy(buffer, inOffset, _chunks[chunkIndex], chunkPos, copied);
inOffset += copied;
left -= copied;
if ((chunkPos + copied) == ChunkSize)
{
chunkIndex++;
chunkPos = 0;
if (chunkIndex == _chunks.Count)
{
_chunks.Add(new byte[ChunkSize]);
}
}
else
{
chunkPos += copied;
}
}
while (left > 0);
_position += count;
if (chunkIndex == (_chunks.Count - 1))
{
if ((chunkIndex > _lastChunkPosIndex) ||
((chunkIndex == _lastChunkPosIndex) && (chunkPos > _lastChunkPos)))
{
_lastChunkPos = chunkPos;
_lastChunkPosIndex = chunkIndex;
}
}
}
/// <summary>
/// Writes a byte to the current position in the stream and advances the position within the stream by one byte.
/// </summary>
/// <param name="value">The byte to write to the stream.</param>
/// <exception cref="T:System.ObjectDisposedException">
/// Methods were called after the stream was closed.
/// </exception>
public override void WriteByte(byte value)
{
CheckDisposed();
int chunkIndex = (int)(_position / ChunkSize);
int chunkPos = (int)(_position % ChunkSize);
if (chunkPos > (ChunkSize - 1)) //changed from (chunkPos >= (ChunkSize - 1))
{
chunkIndex++;
chunkPos = 0;
if (chunkIndex == _chunks.Count)
{
_chunks.Add(new byte[ChunkSize]);
}
}
_chunks[chunkIndex][chunkPos++] = value;
_position++;
if (chunkIndex == (_chunks.Count - 1))
{
if ((chunkIndex > _lastChunkPosIndex) ||
((chunkIndex == _lastChunkPosIndex) && (chunkPos > _lastChunkPos)))
{
_lastChunkPos = chunkPos;
_lastChunkPosIndex = chunkIndex;
}
}
}
/// <summary>
/// Writes to the specified stream.
/// </summary>
/// <param name="stream">The stream.</param>
public void WriteTo(Stream stream)
{
if (stream == null)
throw new ArgumentNullException("stream");
CheckDisposed();
for (int i = 0; i < _chunks.Count; i++)
{
int count = (i == (_chunks.Count - 1)) ? _lastChunkPos : _chunks[i].Length;
stream.Write(_chunks[i], 0, count);
}
}
/// <summary>
/// When overridden in a derived class, gets a value indicating whether the current stream supports reading.
/// </summary>
/// <value></value>
/// <returns>true if the stream supports reading; otherwise, false.
/// </returns>
public override bool CanRead
{
get
{
return true;
}
}
/// <summary>
/// When overridden in a derived class, gets a value indicating whether the current stream supports seeking.
/// </summary>
/// <value></value>
/// <returns>true if the stream supports seeking; otherwise, false.
/// </returns>
public override bool CanSeek
{
get
{
return true;
}
}
/// <summary>
/// When overridden in a derived class, gets a value indicating whether the current stream supports writing.
/// </summary>
/// <value></value>
/// <returns>true if the stream supports writing; otherwise, false.
/// </returns>
public override bool CanWrite
{
get
{
return true;
}
}
/// <summary>
/// When overridden in a derived class, gets the length in bytes of the stream.
/// </summary>
/// <value></value>
/// <returns>
/// A long value representing the length of the stream in bytes.
/// </returns>
/// <exception cref="T:System.ObjectDisposedException">
/// Methods were called after the stream was closed.
/// </exception>
public override long Length
{
get
{
CheckDisposed();
if (_chunks.Count == 0)
return 0;
return (_chunks.Count - 1) * ChunkSize + _lastChunkPos;
}
}
/// <summary>
/// Gets or sets the size of the underlying chunks. Cannot be greater than or equal to 85000.
/// </summary>
/// <value>The chunks size.</value>
public int ChunkSize
{
get
{
return _chunkSize;
}
set
{
if ((value <= 0) || (value >= 85000))
throw new ArgumentOutOfRangeException("value");
_chunkSize = value;
}
}
/// <summary>
/// When overridden in a derived class, gets or sets the position within the current stream.
/// </summary>
/// <value></value>
/// <returns>
/// The current position within the stream.
/// </returns>
/// <exception cref="T:System.ObjectDisposedException">
/// Methods were called after the stream was closed.
/// </exception>
public override long Position
{
get
{
CheckDisposed();
return _position;
}
set
{
CheckDisposed();
if (value < 0)
throw new ArgumentOutOfRangeException("value");
if (value > Length)
throw new ArgumentOutOfRangeException("value");
_position = value;
}
}
}
回答by EricLaw
The Bing team has released RecyclableMemoryStreamand wrote about it here. The benefits they cite are:
Bing 团队已经发布了RecyclableMemoryStream并在此处撰写了相关文章。他们引用的好处是：
- Eliminate Large Object Heap allocations by using pooled buffers
- Incur far fewer gen 2 GCs, and spend far less time paused due to GC
- Avoid memory leaks by having a bounded pool size
- Avoid memory fragmentation
- Provide excellent debuggability
- Provide metrics for performance tracking
- 使用缓冲池消除大对象堆分配
- 产生更少的第 2 代 GC，并且因 GC 而暂停的时间更少
- 通过限制池大小来避免内存泄漏
- 避免内存碎片
- 提供出色的可调试性
- 提供绩效跟踪指标
回答by ryanovic
Another implementation of chunked stream could be considered as a stock MemoryStream replacement. Additionally it allows to allocate a single large byte array on LOH which will be used as a "chunk" pool, shared between all ChunkedStream instances...
分块流的另一种实现可以被认为是一个库存的 MemoryStream 替代品。此外，它还允许在 LOH 上分配一个大字节数组，该数组将用作“块”池，在所有 ChunkedStream 实例之间共享...
