Copyright	(c) Duncan Coutts 2012-2013 Julian Ospald 2022
License	BSD-style
Maintainer	hasufell@posteo.de
Stability	stable
Portability	ghc only
Safe Haskell	Safe-Inferred
Language	Haskell2010

System.OsString.Data.ByteString.Short

Contents

The ShortByteString type
- Memory overhead
- Heap fragmentation
Introducing and eliminating ShortByteStrings
Basic interface
Transforming ShortByteStrings
Reducing ShortByteStrings (folds)
- Special folds
- Generating and unfolding ByteStrings
Substrings
- Breaking strings
Predicates
- Search for arbitrary substrings
Searching ShortByteStrings
- Searching by equality
- Searching with a predicate
Indexing ShortByteStrings
Low level conversions
- Packing CStrings and pointers
- Using ShortByteStrings as CStrings

Description

A compact representation suitable for storing short byte strings in memory.

In typical use cases it can be imported alongside Data.ByteString, e.g.

import qualified Data.ByteString       as B
import qualified Data.ByteString.Short as B
         (ShortByteString, toShort, fromShort)

Other ShortByteString operations clash with Data.ByteString or Prelude functions however, so they should be imported qualified with a different alias e.g.

import qualified Data.ByteString.Short as B.Short

Synopsis

data ShortByteString = SBS ByteArray#
empty :: ShortByteString
singleton :: Word8 -> ShortByteString
pack :: [Word8] -> ShortByteString
unpack :: ShortByteString -> [Word8]
fromShort :: ShortByteString -> ByteString
toShort :: ByteString -> ShortByteString
snoc :: ShortByteString -> Word8 -> ShortByteString
cons :: Word8 -> ShortByteString -> ShortByteString
append :: ShortByteString -> ShortByteString -> ShortByteString
last :: HasCallStack => ShortByteString -> Word8
tail :: HasCallStack => ShortByteString -> ShortByteString
uncons :: ShortByteString -> Maybe (Word8, ShortByteString)
uncons2 :: ShortByteString -> Maybe (Word8, Word8, ShortByteString)
head :: HasCallStack => ShortByteString -> Word8
init :: HasCallStack => ShortByteString -> ShortByteString
unsnoc :: ShortByteString -> Maybe (ShortByteString, Word8)
null :: ShortByteString -> Bool
length :: ShortByteString -> Int
map :: (Word8 -> Word8) -> ShortByteString -> ShortByteString
reverse :: ShortByteString -> ShortByteString
intercalate :: ShortByteString -> [ShortByteString] -> ShortByteString
foldl :: (a -> Word8 -> a) -> a -> ShortByteString -> a
foldl' :: (a -> Word8 -> a) -> a -> ShortByteString -> a
foldl1 :: HasCallStack => (Word8 -> Word8 -> Word8) -> ShortByteString -> Word8
foldl1' :: HasCallStack => (Word8 -> Word8 -> Word8) -> ShortByteString -> Word8
foldr :: (Word8 -> a -> a) -> a -> ShortByteString -> a
foldr' :: (Word8 -> a -> a) -> a -> ShortByteString -> a
foldr1 :: HasCallStack => (Word8 -> Word8 -> Word8) -> ShortByteString -> Word8
foldr1' :: HasCallStack => (Word8 -> Word8 -> Word8) -> ShortByteString -> Word8
all :: (Word8 -> Bool) -> ShortByteString -> Bool
any :: (Word8 -> Bool) -> ShortByteString -> Bool
concat :: [ShortByteString] -> ShortByteString
replicate :: Int -> Word8 -> ShortByteString
unfoldr :: (a -> Maybe (Word8, a)) -> a -> ShortByteString
unfoldrN :: Int -> (a -> Maybe (Word8, a)) -> a -> (ShortByteString, Maybe a)
take :: Int -> ShortByteString -> ShortByteString
takeEnd :: Int -> ShortByteString -> ShortByteString
takeWhileEnd :: (Word8 -> Bool) -> ShortByteString -> ShortByteString
takeWhile :: (Word8 -> Bool) -> ShortByteString -> ShortByteString
drop :: Int -> ShortByteString -> ShortByteString
dropEnd :: Int -> ShortByteString -> ShortByteString
dropWhile :: (Word8 -> Bool) -> ShortByteString -> ShortByteString
dropWhileEnd :: (Word8 -> Bool) -> ShortByteString -> ShortByteString
breakEnd :: (Word8 -> Bool) -> ShortByteString -> (ShortByteString, ShortByteString)
break :: (Word8 -> Bool) -> ShortByteString -> (ShortByteString, ShortByteString)
span :: (Word8 -> Bool) -> ShortByteString -> (ShortByteString, ShortByteString)
spanEnd :: (Word8 -> Bool) -> ShortByteString -> (ShortByteString, ShortByteString)
splitAt :: Int -> ShortByteString -> (ShortByteString, ShortByteString)
split :: Word8 -> ShortByteString -> [ShortByteString]
splitWith :: (Word8 -> Bool) -> ShortByteString -> [ShortByteString]
stripSuffix :: ShortByteString -> ShortByteString -> Maybe ShortByteString
stripPrefix :: ShortByteString -> ShortByteString -> Maybe ShortByteString
isInfixOf :: ShortByteString -> ShortByteString -> Bool
isPrefixOf :: ShortByteString -> ShortByteString -> Bool
isSuffixOf :: ShortByteString -> ShortByteString -> Bool
breakSubstring :: ShortByteString -> ShortByteString -> (ShortByteString, ShortByteString)
elem :: Word8 -> ShortByteString -> Bool
find :: (Word8 -> Bool) -> ShortByteString -> Maybe Word8
filter :: (Word8 -> Bool) -> ShortByteString -> ShortByteString
partition :: (Word8 -> Bool) -> ShortByteString -> (ShortByteString, ShortByteString)
index :: HasCallStack => ShortByteString -> Int -> Word8
indexMaybe :: ShortByteString -> Int -> Maybe Word8
(!?) :: ShortByteString -> Int -> Maybe Word8
elemIndex :: Word8 -> ShortByteString -> Maybe Int
elemIndices :: Word8 -> ShortByteString -> [Int]
count :: Word8 -> ShortByteString -> Int
findIndex :: (Word8 -> Bool) -> ShortByteString -> Maybe Int
findIndices :: (Word8 -> Bool) -> ShortByteString -> [Int]
packCString :: CString -> IO ShortByteString
packCStringLen :: CStringLen -> IO ShortByteString
useAsCString :: ShortByteString -> (CString -> IO a) -> IO a
useAsCStringLen :: ShortByteString -> (CStringLen -> IO a) -> IO a

The `ShortByteString` type

data ShortByteString #

Constructors

SBS ByteArray#

Instances

Instances details

Data ShortByteString
Instance details Defined in Data.ByteString.Short.Internal Methods gfoldl :: (forall d b. Data d => c (d -> b) -> d -> c b) -> (forall g. g -> c g) -> ShortByteString -> c ShortByteString gunfold :: (forall b r. Data b => c (b -> r) -> c r) -> (forall r. r -> c r) -> Constr -> c ShortByteString toConstr :: ShortByteString -> Constr dataTypeOf :: ShortByteString -> DataType dataCast1 :: Typeable t => (forall d. Data d => c (t d)) -> Maybe (c ShortByteString) dataCast2 :: Typeable t => (forall d e. (Data d, Data e) => c (t d e)) -> Maybe (c ShortByteString) gmapT :: (forall b. Data b => b -> b) -> ShortByteString -> ShortByteString gmapQl :: (r -> r' -> r) -> r -> (forall d. Data d => d -> r') -> ShortByteString -> r gmapQr :: forall r r'. (r' -> r -> r) -> r -> (forall d. Data d => d -> r') -> ShortByteString -> r gmapQ :: (forall d. Data d => d -> u) -> ShortByteString -> [u] gmapQi :: Int -> (forall d. Data d => d -> u) -> ShortByteString -> u gmapM :: Monad m => (forall d. Data d => d -> m d) -> ShortByteString -> m ShortByteString gmapMp :: MonadPlus m => (forall d. Data d => d -> m d) -> ShortByteString -> m ShortByteString gmapMo :: MonadPlus m => (forall d. Data d => d -> m d) -> ShortByteString -> m ShortByteString
IsString ShortByteString
Instance details Defined in Data.ByteString.Short.Internal Methods fromString :: String -> ShortByteString
Monoid ShortByteString
Instance details Defined in Data.ByteString.Short.Internal Methods mempty :: ShortByteString mappend :: ShortByteString -> ShortByteString -> ShortByteString mconcat :: [ShortByteString] -> ShortByteString
Semigroup ShortByteString
Instance details Defined in Data.ByteString.Short.Internal Methods (<>) :: ShortByteString -> ShortByteString -> ShortByteString sconcat :: NonEmpty ShortByteString -> ShortByteString stimes :: Integral b => b -> ShortByteString -> ShortByteString
IsList ShortByteString
Instance details Defined in Data.ByteString.Short.Internal Associated Types type Item ShortByteString Methods fromList :: [Item ShortByteString] -> ShortByteString fromListN :: Int -> [Item ShortByteString] -> ShortByteString toList :: ShortByteString -> [Item ShortByteString]
Read ShortByteString
Instance details Defined in Data.ByteString.Short.Internal Methods readsPrec :: Int -> ReadS ShortByteString readList :: ReadS [ShortByteString] readPrec :: ReadPrec ShortByteString readListPrec :: ReadPrec [ShortByteString]
Show ShortByteString
Instance details Defined in Data.ByteString.Short.Internal Methods showsPrec :: Int -> ShortByteString -> ShowS show :: ShortByteString -> String showList :: [ShortByteString] -> ShowS
NFData ShortByteString
Instance details Defined in Data.ByteString.Short.Internal Methods rnf :: ShortByteString -> ()
Eq ShortByteString
Instance details Defined in Data.ByteString.Short.Internal Methods (==) :: ShortByteString -> ShortByteString -> Bool (/=) :: ShortByteString -> ShortByteString -> Bool
Ord ShortByteString
Instance details Defined in Data.ByteString.Short.Internal Methods compare :: ShortByteString -> ShortByteString -> Ordering (<) :: ShortByteString -> ShortByteString -> Bool (<=) :: ShortByteString -> ShortByteString -> Bool (>) :: ShortByteString -> ShortByteString -> Bool (>=) :: ShortByteString -> ShortByteString -> Bool max :: ShortByteString -> ShortByteString -> ShortByteString min :: ShortByteString -> ShortByteString -> ShortByteString
Lift ShortByteString
Instance details Defined in Data.ByteString.Short.Internal Methods lift :: Quote m => ShortByteString -> m Exp liftTyped :: forall (m :: Type -> Type). Quote m => ShortByteString -> Code m ShortByteString
type Item ShortByteString
Instance details Defined in Data.ByteString.Short.Internal type Item ShortByteString = Word8

Memory overhead

With GHC, the memory overheads are as follows, expressed in words and in bytes (words are 4 and 8 bytes on 32 or 64bit machines respectively).

ByteString unshared: 8 words; 32 or 64 bytes.
ByteString shared substring: 4 words; 16 or 32 bytes.
ShortByteString: 4 words; 16 or 32 bytes.

For the string data itself, both ShortByteString and ByteString use one byte per element, rounded up to the nearest word. For example, including the overheads, a length 10 ShortByteString would take 16 + 12 = 28 bytes on a 32bit platform and 32 + 16 = 48 bytes on a 64bit platform.

These overheads can all be reduced by 1 word (4 or 8 bytes) when the ShortByteString or ByteString is unpacked into another constructor.

For example:

data ThingId = ThingId {-# UNPACK #-} !Int
                       {-# UNPACK #-} !ShortByteString

This will take 1 + 1 + 3 words (the ThingId constructor + unpacked Int + unpacked ShortByteString), plus the words for the string data.

Heap fragmentation

With GHC, the ByteString representation uses pinned memory, meaning it cannot be moved by the GC. This is usually the right thing to do for larger strings, but for small strings using pinned memory can lead to heap fragmentation which wastes space. The ShortByteString type (and the Text type from the text package) use unpinned memory so they do not contribute to heap fragmentation. In addition, with GHC, small unpinned strings are allocated in the same way as normal heap allocations, rather than in a separate pinned area.