consistent or non-consistent memory as it sees fit. By using this API,
you are guaranteeing to the platform that you have all the correct and
necessary sync points for this memory in the driver.
+
+DMA_ATTR_NO_KERNEL_MAPPING
+--------------------------
+
+DMA_ATTR_NO_KERNEL_MAPPING lets the platform to avoid creating a kernel
+virtual mapping for the allocated buffer. On some architectures creating
+such mapping is non-trivial task and consumes very limited resources
+(like kernel virtual address space or dma consistent address space).
+Buffers allocated with this attribute can be only passed to user space
+by calling dma_mmap_attrs(). By using this API, you are guaranteeing
+that you won't dereference the pointer returned by dma_alloc_attr(). You
+can threat it as a cookie that must be passed to dma_mmap_attrs() and
+dma_free_attrs(). Make sure that both of these also get this attribute
+set on each call.
+
+Since it is optional for platforms to implement
+DMA_ATTR_NO_KERNEL_MAPPING, those that do not will simply ignore the
+attribute and exhibit default behavior.
+
+DMA_ATTR_SKIP_CPU_SYNC
+----------------------
+
+By default dma_map_{single,page,sg} functions family transfer a given
+buffer from CPU domain to device domain. Some advanced use cases might
+require sharing a buffer between more than one device. This requires
+having a mapping created separately for each device and is usually
+performed by calling dma_map_{single,page,sg} function more than once
+for the given buffer with device pointer to each device taking part in
+the buffer sharing. The first call transfers a buffer from 'CPU' domain
+to 'device' domain, what synchronizes CPU caches for the given region
+(usually it means that the cache has been flushed or invalidated
+depending on the dma direction). However, next calls to
+dma_map_{single,page,sg}() for other devices will perform exactly the
+same sychronization operation on the CPU cache. CPU cache sychronization
+might be a time consuming operation, especially if the buffers are
+large, so it is highly recommended to avoid it if possible.
+DMA_ATTR_SKIP_CPU_SYNC allows platform code to skip synchronization of
+the CPU cache for the given buffer assuming that it has been already
+transferred to 'device' domain. This attribute can be also used for
+dma_unmap_{single,page,sg} functions family to force buffer to stay in
+device domain after releasing a mapping for it. Use this attribute with
+care!
<listitem>
<para>Selection API. <xref linkend="selection-api" /></para>
</listitem>
+ <listitem>
+ <para>Importing DMABUF file descriptors as a new IO method described
+ in <xref linkend="dmabuf" />.</para>
+ </listitem>
+ <listitem>
+ <para>Exporting DMABUF files using &VIDIOC-EXPBUF; ioctl.</para>
+ </listitem>
</itemizedlist>
</section>
</footnote></para>
</section>
+ <section id="dmabuf">
+ <title>Streaming I/O (DMA buffer importing)</title>
+
+ <note>
+ <title>Experimental</title>
+ <para>This is an <link linkend="experimental"> experimental </link>
+ interface and may change in the future.</para>
+ </note>
+
+<para>The DMABUF framework provides a generic mean for sharing buffers between
+ multiple devices. Device drivers that support DMABUF can export a DMA buffer
+to userspace as a file descriptor (known as the exporter role), import a DMA
+buffer from userspace using a file descriptor previously exported for a
+different or the same device (known as the importer role), or both. This
+section describes the DMABUF importer role API in V4L2.</para>
+
+ <para>Refer to <link linked="vidioc-expbuf"> DMABUF exporting </link> for
+details about exporting a V4L2 buffers as DMABUF file descriptors.</para>
+
+<para>Input and output devices support the streaming I/O method when the
+<constant>V4L2_CAP_STREAMING</constant> flag in the
+<structfield>capabilities</structfield> field of &v4l2-capability; returned by
+the &VIDIOC-QUERYCAP; ioctl is set. Whether importing DMA buffers through
+DMABUF file descriptors is supported is determined by calling the
+&VIDIOC-REQBUFS; ioctl with the memory type set to
+<constant>V4L2_MEMORY_DMABUF</constant>.</para>
+
+ <para>This I/O method is dedicated for sharing DMA buffers between V4L and
+other APIs. Buffers (planes) are allocated by a driver on behalf of the
+application, and exported to the application as file descriptors using an API
+specific to the allocator driver. Only those file descriptor are exchanged,
+these files and meta-information are passed in &v4l2-buffer; (or in
+&v4l2-plane; in the multi-planar API case). The driver must be switched into
+DMABUF I/O mode by calling the &VIDIOC-REQBUFS; with the desired buffer type.
+No buffers (planes) are allocated beforehand, consequently they are not indexed
+and cannot be queried like mapped buffers with the
+<constant>VIDIOC_QUERYBUF</constant> ioctl.</para>
+
+ <example>
+ <title>Initiating streaming I/O with DMABUF file descriptors</title>
+
+ <programlisting>
+&v4l2-requestbuffers; reqbuf;
+
+memset (&reqbuf, 0, sizeof (reqbuf));
+reqbuf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+reqbuf.memory = V4L2_MEMORY_DMABUF;
+
+if (ioctl (fd, &VIDIOC-REQBUFS;, &reqbuf) == -1) {
+ if (errno == EINVAL)
+ printf ("Video capturing or DMABUF streaming is not supported\n");
+ else
+ perror ("VIDIOC_REQBUFS");
+
+ exit (EXIT_FAILURE);
+}
+ </programlisting>
+ </example>
+
+ <para>Buffer (plane) file is passed on the fly with the &VIDIOC-QBUF;
+ioctl. In case of multiplanar buffers, every plane can be associated with a
+different DMABUF descriptor.Although buffers are commonly cycled, applications
+can pass different DMABUF descriptor at each <constant>VIDIOC_QBUF</constant>
+call.</para>
+
+ <example>
+ <title>Queueing DMABUF using single plane API</title>
+
+ <programlisting>
+int buffer_queue(int v4lfd, int index, int dmafd)
+{
+ &v4l2-buffer; buf;
+
+ memset(&buf, 0, sizeof buf);
+ buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+ buf.memory = V4L2_MEMORY_DMABUF;
+ buf.index = index;
+ buf.m.fd = dmafd;
+
+ if (ioctl (v4lfd, &VIDIOC-QBUF;, &buf) == -1) {
+ perror ("VIDIOC_QBUF");
+ return -1;
+ }
+
+ return 0;
+}
+ </programlisting>
+ </example>
+
+ <example>
+ <title>Queueing DMABUF using multi plane API</title>
+
+ <programlisting>
+int buffer_queue_mp(int v4lfd, int index, int dmafd[], int n_planes)
+{
+ &v4l2-buffer; buf;
+ &v4l2-plane; planes[VIDEO_MAX_PLANES];
+ int i;
+
+ memset(&buf, 0, sizeof buf);
+ buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+ buf.memory = V4L2_MEMORY_DMABUF;
+ buf.index = index;
+ buf.m.planes = planes;
+ buf.length = n_planes;
+
+ memset(&planes, 0, sizeof planes);
+
+ for (i = 0; i < n_planes; ++i)
+ buf.m.planes[i].m.fd = dmafd[i];
+
+ if (ioctl (v4lfd, &VIDIOC-QBUF;, &buf) == -1) {
+ perror ("VIDIOC_QBUF");
+ return -1;
+ }
+
+ return 0;
+}
+ </programlisting>
+ </example>
+
+ <para>Filled or displayed buffers are dequeued with the
+&VIDIOC-DQBUF; ioctl. The driver can unlock the buffer at any
+time between the completion of the DMA and this ioctl. The memory is
+also unlocked when &VIDIOC-STREAMOFF; is called, &VIDIOC-REQBUFS;, or
+when the device is closed.</para>
+
+ <para>For capturing applications it is customary to enqueue a
+number of empty buffers, to start capturing and enter the read loop.
+Here the application waits until a filled buffer can be dequeued, and
+re-enqueues the buffer when the data is no longer needed. Output
+applications fill and enqueue buffers, when enough buffers are stacked
+up output is started. In the write loop, when the application
+runs out of free buffers it must wait until an empty buffer can be
+dequeued and reused. Two methods exist to suspend execution of the
+application until one or more buffers can be dequeued. By default
+<constant>VIDIOC_DQBUF</constant> blocks when no buffer is in the
+outgoing queue. When the <constant>O_NONBLOCK</constant> flag was
+given to the &func-open; function, <constant>VIDIOC_DQBUF</constant>
+returns immediately with an &EAGAIN; when no buffer is available. The
+&func-select; or &func-poll; function are always available.</para>
+
+ <para>To start and stop capturing or output applications call the
+&VIDIOC-STREAMON; and &VIDIOC-STREAMOFF; ioctls. Note that
+<constant>VIDIOC_STREAMOFF</constant> removes all buffers from both queues and
+unlocks all buffers as a side effect. Since there is no notion of doing
+anything "now" on a multitasking system, if an application needs to synchronize
+with another event it should examine the &v4l2-buffer;
+<structfield>timestamp</structfield> of captured buffers, or set the field
+before enqueuing buffers for output.</para>
+
+ <para>Drivers implementing DMABUF importing I/O must support the
+<constant>VIDIOC_REQBUFS</constant>, <constant>VIDIOC_QBUF</constant>,
+<constant>VIDIOC_DQBUF</constant>, <constant>VIDIOC_STREAMON</constant> and
+<constant>VIDIOC_STREAMOFF</constant> ioctl, the <function>select()</function>
+and <function>poll()</function> function.</para>
+
+ </section>
+
<section id="async">
<title>Asynchronous I/O</title>
in the <structfield>length</structfield> field of this
<structname>v4l2_buffer</structname> structure.</entry>
</row>
+ <row>
+ <entry></entry>
+ <entry>int</entry>
+ <entry><structfield>fd</structfield></entry>
+ <entry>For the single-plane API and when
+<structfield>memory</structfield> is <constant>V4L2_MEMORY_DMABUF</constant> this
+is the file descriptor associated with a DMABUF buffer.</entry>
+ </row>
<row>
<entry>__u32</entry>
<entry><structfield>length</structfield></entry>
pointer to the memory allocated for this plane by an application.
</entry>
</row>
+ <row>
+ <entry></entry>
+ <entry>int</entry>
+ <entry><structfield>fd</structfield></entry>
+ <entry>When the memory type in the containing &v4l2-buffer; is
+ <constant>V4L2_MEMORY_DMABUF</constant>, this is a file
+ descriptor associated with a DMABUF buffer, similar to the
+ <structfield>fd</structfield> field in &v4l2-buffer;.</entry>
+ </row>
<row>
<entry>__u32</entry>
<entry><structfield>data_offset</structfield></entry>
<entry>3</entry>
<entry>[to do]</entry>
</row>
+ <row>
+ <entry><constant>V4L2_MEMORY_DMABUF</constant></entry>
+ <entry>4</entry>
+ <entry>The buffer is used for <link linkend="dmabuf">DMA shared
+buffer</link> I/O.</entry>
+ </row>
</tbody>
</tgroup>
</table>
&sub-log-status;
&sub-overlay;
&sub-qbuf;
+ &sub-expbuf;
&sub-querybuf;
&sub-querycap;
&sub-queryctrl;
<entry>&v4l2-memory;</entry>
<entry><structfield>memory</structfield></entry>
<entry>Applications set this field to
-<constant>V4L2_MEMORY_MMAP</constant> or
-<constant>V4L2_MEMORY_USERPTR</constant>.</entry>
+<constant>V4L2_MEMORY_MMAP</constant>,
+<constant>V4L2_MEMORY_DMABUF</constant> or
+<constant>V4L2_MEMORY_USERPTR</constant>. See <xref linkend="v4l2-memory"
+/></entry>
</row>
<row>
<entry>&v4l2-format;</entry>
--- /dev/null
+<refentry id="vidioc-expbuf">
+
+ <refmeta>
+ <refentrytitle>ioctl VIDIOC_EXPBUF</refentrytitle>
+ &manvol;
+ </refmeta>
+
+ <refnamediv>
+ <refname>VIDIOC_EXPBUF</refname>
+ <refpurpose>Export a buffer as a DMABUF file descriptor.</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <funcsynopsis>
+ <funcprototype>
+ <funcdef>int <function>ioctl</function></funcdef>
+ <paramdef>int <parameter>fd</parameter></paramdef>
+ <paramdef>int <parameter>request</parameter></paramdef>
+ <paramdef>struct v4l2_exportbuffer *<parameter>argp</parameter></paramdef>
+ </funcprototype>
+ </funcsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1>
+ <title>Arguments</title>
+
+ <variablelist>
+ <varlistentry>
+ <term><parameter>fd</parameter></term>
+ <listitem>
+ <para>&fd;</para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><parameter>request</parameter></term>
+ <listitem>
+ <para>VIDIOC_EXPBUF</para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><parameter>argp</parameter></term>
+ <listitem>
+ <para></para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </refsect1>
+
+ <refsect1>
+ <title>Description</title>
+
+ <note>
+ <title>Experimental</title>
+ <para>This is an <link linkend="experimental"> experimental </link>
+ interface and may change in the future.</para>
+ </note>
+
+<para>This ioctl is an extension to the <link linkend="mmap">memory
+mapping</link> I/O method therefore it is available only for
+<constant>V4L2_MEMORY_MMAP</constant> buffers. It can be used to export a
+buffer as DMABUF file at any time after buffers have been allocated with the
+&VIDIOC-REQBUFS; ioctl.</para>
+
+<para>Prior to exporting an application calls <link
+linkend="vidioc-querybuf">VIDIOC_QUERYBUF</link> to obtain memory offsets. When
+using the <link linkend="planar-apis">multi-planar API</link> every plane has
+own offset.</para>
+
+<para>To export a buffer, the application fills &v4l2-exportbuffer;. The
+<structfield> mem_offset </structfield> field is set to the offset obtained
+from <constant> VIDIOC_QUERYBUF </constant>. Additional flags may be posted in
+the <structfield> flags </structfield> field. Refer to manual for open syscall
+for details. Currently only O_CLOEXEC is guaranteed to be supported. All other
+fields must be set to zero. In a case of multi-planar API, every plane is
+exported separately using multiple <constant> VIDIOC_EXPBUF </constant>
+calls.</para>
+
+<para> After calling <constant>VIDIOC_EXPBUF</constant> the <structfield> fd
+</structfield> field will be set by a driver. This is a DMABUF file
+descriptor. The application may pass it to other API. Refer to <link
+linkend="dmabuf">DMABUF importing</link> for details about importing DMABUF
+files into V4L2 nodes. A developer is encouraged to close a DMABUF file when it
+is no longer used. </para>
+
+ </refsect1>
+ <refsect1>
+ <section>
+ <title>Examples</title>
+
+ <example>
+ <title>Exporting a buffer.</title>
+ <programlisting>
+int buffer_export(int v4lfd, &v4l2-buf-type; bt, int index, int *dmafd)
+{
+ &v4l2-buffer; buf;
+ &v4l2-exportbuffer; expbuf;
+
+ memset(&buf, 0, sizeof buf);
+ buf.type = bt;
+ buf.memory = V4L2_MEMORY_MMAP;
+ buf.index = index;
+
+ if (ioctl (v4lfd, &VIDIOC-QUERYBUF;, &buf) == -1) {
+ perror ("VIDIOC_QUERYBUF");
+ return -1;
+ }
+
+ memset(&expbuf, 0, sizeof expbuf);
+ expbuf.mem_offset = buf.m.offset;
+ if (ioctl (v4lfd, &VIDIOC-EXPBUF;, &expbuf) == -1) {
+ perror ("VIDIOC_EXPBUF");
+ return -1;
+ }
+
+ *dmafd = expbuf.fd;
+
+ return 0;
+}
+ </programlisting>
+ </example>
+
+ <example>
+ <title>Exporting a buffer using multi plane API.</title>
+ <programlisting>
+int buffer_export_mp(int v4lfd, &v4l2-buf-type; bt, int index,
+ int dmafd[], int n_planes)
+{
+ &v4l2-buffer; buf;
+ &v4l2-plane; planes[VIDEO_MAX_PLANES];
+ int i;
+
+ memset(&buf, 0, sizeof buf);
+ buf.type = bt;
+ buf.memory = V4L2_MEMORY_MMAP;
+ buf.index = index;
+ buf.m.planes = planes;
+ buf.length = n_planes;
+ memset(&planes, 0, sizeof planes);
+
+ if (ioctl (v4lfd, &VIDIOC-QUERYBUF;, &buf) == -1) {
+ perror ("VIDIOC_QUERYBUF");
+ return -1;
+ }
+
+ for (i = 0; i < n_planes; ++i) {
+ &v4l2-exportbuffer; expbuf;
+
+ memset(&expbuf, 0, sizeof expbuf);
+ expbuf.mem_offset = plane[i].m.offset;
+ if (ioctl (v4lfd, &VIDIOC-EXPBUF;, &expbuf) == -1) {
+ perror ("VIDIOC_EXPBUF");
+ while (i)
+ close(dmafd[--i]);
+ return -1;
+ }
+ dmafd[i] = expbuf.fd;
+ }
+
+ return 0;
+}
+ </programlisting>
+ </example>
+ </section>
+ </refsect1>
+
+ <refsect1>
+ <table pgwide="1" frame="none" id="v4l2-exportbuffer">
+ <title>struct <structname>v4l2_exportbuffer</structname></title>
+ <tgroup cols="3">
+ &cs-str;
+ <tbody valign="top">
+ <row>
+ <entry>__u32</entry>
+ <entry><structfield>fd</structfield></entry>
+ <entry>The DMABUF file descriptor associated with a buffer. Set by
+ a driver.</entry>
+ </row>
+ <row>
+ <entry>__u32</entry>
+ <entry><structfield>reserved0</structfield></entry>
+ <entry>Reserved field for future use. Must be set to zero.</entry>
+ </row>
+ <row>
+ <entry>__u32</entry>
+ <entry><structfield>mem_offset</structfield></entry>
+ <entry>Buffer memory offset as returned by <constant>
+VIDIOC_QUERYBUF </constant> in &v4l2-buffer;<structfield> ::m.offset
+</structfield> (for single-plane formats) or &v4l2-plane;<structfield>
+::m.offset </structfield> (for multi-planar formats)</entry>
+ </row>
+ <row>
+ <entry>__u32</entry>
+ <entry><structfield>flags</structfield></entry>
+ <entry>Flags for newly created file, currently only <constant>
+O_CLOEXEC </constant> is supported, refer to manual of open syscall for more
+details.</entry>
+ </row>
+ <row>
+ <entry>__u32</entry>
+ <entry><structfield>reserved[12]</structfield></entry>
+ <entry>Reserved field for future use. Must be set to zero.</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ </refsect1>
+
+ <refsect1>
+ &return-value;
+ <variablelist>
+ <varlistentry>
+ <term><errorcode>EINVAL</errorcode></term>
+ <listitem>
+ <para>A queue is not in MMAP mode or DMABUF exporting is not
+supported or <structfield> flag </structfield> or <structfield> mem_offset
+</structfield> fields are invalid.</para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </refsect1>
+
+</refentry>
dequeued, until the &VIDIOC-STREAMOFF; or &VIDIOC-REQBUFS; ioctl is
called, or until the device is closed.</para>
+ <para>To enqueue a <link linkend="dmabuf">DMABUF</link> buffer applications
+set the <structfield>memory</structfield> field to
+<constant>V4L2_MEMORY_DMABUF</constant> and the <structfield>m.fd</structfield>
+to a file descriptor associated with a DMABUF buffer. When the multi-planar API is
+used and <structfield>m.fd</structfield> of the passed array of &v4l2-plane;
+have to be used instead. When <constant>VIDIOC_QBUF</constant> is called with a
+pointer to this structure the driver sets the
+<constant>V4L2_BUF_FLAG_QUEUED</constant> flag and clears the
+<constant>V4L2_BUF_FLAG_MAPPED</constant> and
+<constant>V4L2_BUF_FLAG_DONE</constant> flags in the
+<structfield>flags</structfield> field, or it returns an error code. This
+ioctl locks the buffer. Buffers remain locked until dequeued,
+until the &VIDIOC-STREAMOFF; or &VIDIOC-REQBUFS; ioctl is called, or until the
+device is closed.</para>
+
<para>Applications call the <constant>VIDIOC_DQBUF</constant>
ioctl to dequeue a filled (capturing) or displayed (output) buffer
from the driver's outgoing queue. They just set the
<refsect1>
<title>Description</title>
- <para>This ioctl is used to initiate <link linkend="mmap">memory
-mapped</link> or <link linkend="userp">user pointer</link>
-I/O. Memory mapped buffers are located in device memory and must be
-allocated with this ioctl before they can be mapped into the
-application's address space. User buffers are allocated by
-applications themselves, and this ioctl is merely used to switch the
-driver into user pointer I/O mode and to setup some internal structures.</para>
+<para>This ioctl is used to initiate <link linkend="mmap">memory mapped</link>,
+<link linkend="userp">user pointer</link> or <link
+linkend="dmabuf">DMABUF</link> based I/O. Memory mapped buffers are located in
+device memory and must be allocated with this ioctl before they can be mapped
+into the application's address space. User buffers are allocated by
+applications themselves, and this ioctl is merely used to switch the driver
+into user pointer I/O mode and to setup some internal structures.
+Similarly, DMABUF buffers are allocated by applications through a device
+driver, and this ioctl only configures the driver into DMABUF I/O mode without
+performing any direct allocation.</para>
- <para>To allocate device buffers applications initialize all
-fields of the <structname>v4l2_requestbuffers</structname> structure.
-They set the <structfield>type</structfield> field to the respective
-stream or buffer type, the <structfield>count</structfield> field to
-the desired number of buffers, <structfield>memory</structfield>
-must be set to the requested I/O method and the <structfield>reserved</structfield> array
-must be zeroed. When the ioctl
-is called with a pointer to this structure the driver will attempt to allocate
-the requested number of buffers and it stores the actual number
-allocated in the <structfield>count</structfield> field. It can be
-smaller than the number requested, even zero, when the driver runs out
-of free memory. A larger number is also possible when the driver requires
-more buffers to function correctly. For example video output requires at least two buffers,
-one displayed and one filled by the application.</para>
+ <para>To allocate device buffers applications initialize all fields of the
+<structname>v4l2_requestbuffers</structname> structure. They set the
+<structfield>type</structfield> field to the respective stream or buffer type,
+the <structfield>count</structfield> field to the desired number of buffers,
+<structfield>memory</structfield> must be set to the requested I/O method and
+the <structfield>reserved</structfield> array must be zeroed. When the ioctl is
+called with a pointer to this structure the driver will attempt to allocate the
+requested number of buffers and it stores the actual number allocated in the
+<structfield>count</structfield> field. It can be smaller than the number
+requested, even zero, when the driver runs out of free memory. A larger number
+is also possible when the driver requires more buffers to function correctly.
+For example video output requires at least two buffers, one displayed and one
+filled by the application.</para>
<para>When the I/O method is not supported the ioctl
returns an &EINVAL;.</para>
<entry>&v4l2-memory;</entry>
<entry><structfield>memory</structfield></entry>
<entry>Applications set this field to
-<constant>V4L2_MEMORY_MMAP</constant> or
-<constant>V4L2_MEMORY_USERPTR</constant>.</entry>
+<constant>V4L2_MEMORY_MMAP</constant>,
+<constant>V4L2_MEMORY_DMABUF</constant> or
+<constant>V4L2_MEMORY_USERPTR</constant>. See <xref linkend="v4l2-memory"
+/>.</entry>
</row>
<row>
<entry>__u32</entry>
--- /dev/null
+* Samsung SPI Controller
+
+The Samsung SPI controller is used to interface with various devices such as flash
+and display controllers using the SPI communication interface.
+
+Required SoC Specific Properties:
+
+- compatible: should be one of the following.
+ - samsung,s3c2443-spi: for s3c2443, s3c2416 and s3c2450 platforms
+ - samsung,s3c6410-spi: for s3c6410 platforms
+ - samsung,s5p6440-spi: for s5p6440 and s5p6450 platforms
+ - samsung,s5pv210-spi: for s5pv210 and s5pc110 platforms
+ - samsung,exynos4210-spi: for exynos4 and exynos5 platforms
+
+- reg: physical base address of the controller and length of memory mapped
+ region.
+
+- interrupts: The interrupt number to the cpu. The interrupt specifier format
+ depends on the interrupt controller.
+
+- tx-dma-channel: The dma channel specifier for tx operations. The format of
+ the dma specifier depends on the dma controller.
+
+- rx-dma-channel: The dma channel specifier for rx operations. The format of
+ the dma specifier depends on the dma controller.
+
+Required Board Specific Properties:
+
+- #address-cells: should be 1.
+- #size-cells: should be 0.
+- gpios: The gpio specifier for clock, mosi and miso interface lines (in the
+ order specified). The format of the gpio specifier depends on the gpio
+ controller.
+
+Optional Board Specific Properties:
+
+- samsung,spi-src-clk: If the spi controller includes a internal clock mux to
+ select the clock source for the spi bus clock, this property can be used to
+ indicate the clock to be used for driving the spi bus clock. If not specified,
+ the clock number 0 is used as default.
+
+- num-cs: Specifies the number of chip select lines supported. If
+ not specified, the default number of chip select lines is set to 1.
+
+SPI Controller specific data in SPI slave nodes:
+
+- The spi slave nodes should provide the following information which is required
+ by the spi controller.
+
+ - cs-gpio: A gpio specifier that specifies the gpio line used as
+ the slave select line by the spi controller. The format of the gpio
+ specifier depends on the gpio controller.
+
+ - samsung,spi-feedback-delay: The sampling phase shift to be applied on the
+ miso line (to account for any lag in the miso line). The following are the
+ valid values.
+
+ - 0: No phase shift.
+ - 1: 90 degree phase shift sampling.
+ - 2: 180 degree phase shift sampling.
+ - 3: 270 degree phase shift sampling.
+
+Aliases:
+
+- All the SPI controller nodes should be represented in the aliases node using
+ the following format 'spi{n}' where n is a unique number for the alias.
+
+
+Example:
+
+- SoC Specific Portion:
+
+ spi_0: spi@12d20000 {
+ compatible = "samsung,exynos4210-spi";
+ reg = <0x12d20000 0x100>;
+ interrupts = <0 66 0>;
+ tx-dma-channel = <&pdma0 5>;
+ rx-dma-channel = <&pdma0 4>;
+ };
+
+- Board Specific Portion:
+
+ spi_0: spi@12d20000 {
+ #address-cells = <1>;
+ #size-cells = <0>;
+ gpios = <&gpa2 4 2 3 0>,
+ <&gpa2 6 2 3 0>,
+ <&gpa2 7 2 3 0>;
+
+ w25q80bw@0 {
+ #address-cells = <1>;
+ #size-cells = <1>;
+ compatible = "w25x80";
+ reg = <0>;
+ spi-max-frequency = <10000>;
+
+ controller-data {
+ cs-gpio = <&gpa2 5 1 0 3>;
+ samsung,spi-feedback-delay = <0>;
+ };
+
+ partition@0 {
+ label = "U-Boot";
+ reg = <0x0 0x40000>;
+ read-only;
+ };
+
+ partition@40000 {
+ label = "Kernel";
+ reg = <0x40000 0xc0000>;
+ };
+ };
+ };
in memory, mapped into its own address space, so it can access the same area
of memory.
-*IMPORTANT*: [see https://lkml.org/lkml/2011/12/20/211 for more details]
-For this first version, A buffer shared using the dma_buf sharing API:
-- *may* be exported to user space using "mmap" *ONLY* by exporter, outside of
- this framework.
-- with this new iteration of the dma-buf api cpu access from the kernel has been
- enable, see below for the details.
-
dma-buf operations for device dma only
--------------------------------------
Note that these calls need to always succeed. The exporter needs to complete
any preparations that might fail in begin_cpu_access.
+ For some cases the overhead of kmap can be too high, a vmap interface
+ is introduced. This interface should be used very carefully, as vmalloc
+ space is a limited resources on many architectures.
+
+ Interfaces:
+ void *dma_buf_vmap(struct dma_buf *dmabuf)
+ void dma_buf_vunmap(struct dma_buf *dmabuf, void *vaddr)
+
+ The vmap call can fail if there is no vmap support in the exporter, or if it
+ runs out of vmalloc space. Fallback to kmap should be implemented.
+
3. Finish access
When the importer is done accessing the range specified in begin_cpu_access,
enum dma_data_direction dir);
+Direct Userspace Access/mmap Support
+------------------------------------
+
+Being able to mmap an export dma-buf buffer object has 2 main use-cases:
+- CPU fallback processing in a pipeline and
+- supporting existing mmap interfaces in importers.
+
+1. CPU fallback processing in a pipeline
+
+ In many processing pipelines it is sometimes required that the cpu can access
+ the data in a dma-buf (e.g. for thumbnail creation, snapshots, ...). To avoid
+ the need to handle this specially in userspace frameworks for buffer sharing
+ it's ideal if the dma_buf fd itself can be used to access the backing storage
+ from userspace using mmap.
+
+ Furthermore Android's ION framework already supports this (and is otherwise
+ rather similar to dma-buf from a userspace consumer side with using fds as
+ handles, too). So it's beneficial to support this in a similar fashion on
+ dma-buf to have a good transition path for existing Android userspace.
+
+ No special interfaces, userspace simply calls mmap on the dma-buf fd.
+
+2. Supporting existing mmap interfaces in exporters
+
+ Similar to the motivation for kernel cpu access it is again important that
+ the userspace code of a given importing subsystem can use the same interfaces
+ with a imported dma-buf buffer object as with a native buffer object. This is
+ especially important for drm where the userspace part of contemporary OpenGL,
+ X, and other drivers is huge, and reworking them to use a different way to
+ mmap a buffer rather invasive.
+
+ The assumption in the current dma-buf interfaces is that redirecting the
+ initial mmap is all that's needed. A survey of some of the existing
+ subsystems shows that no driver seems to do any nefarious thing like syncing
+ up with outstanding asynchronous processing on the device or allocating
+ special resources at fault time. So hopefully this is good enough, since
+ adding interfaces to intercept pagefaults and allow pte shootdowns would
+ increase the complexity quite a bit.
+
+ Interface:
+ int dma_buf_mmap(struct dma_buf *, struct vm_area_struct *,
+ unsigned long);
+
+ If the importing subsystem simply provides a special-purpose mmap call to set
+ up a mapping in userspace, calling do_mmap with dma_buf->file will equally
+ achieve that for a dma-buf object.
+
+3. Implementation notes for exporters
+
+ Because dma-buf buffers have invariant size over their lifetime, the dma-buf
+ core checks whether a vma is too large and rejects such mappings. The
+ exporter hence does not need to duplicate this check.
+
+ Because existing importing subsystems might presume coherent mappings for
+ userspace, the exporter needs to set up a coherent mapping. If that's not
+ possible, it needs to fake coherency by manually shooting down ptes when
+ leaving the cpu domain and flushing caches at fault time. Note that all the
+ dma_buf files share the same anon inode, hence the exporter needs to replace
+ the dma_buf file stored in vma->vm_file with it's own if pte shootdown is
+ requred. This is because the kernel uses the underlying inode's address_space
+ for vma tracking (and hence pte tracking at shootdown time with
+ unmap_mapping_range).
+
+ If the above shootdown dance turns out to be too expensive in certain
+ scenarios, we can extend dma-buf with a more explicit cache tracking scheme
+ for userspace mappings. But the current assumption is that using mmap is
+ always a slower path, so some inefficiencies should be acceptable.
+
+ Exporters that shoot down mappings (for any reasons) shall not do any
+ synchronization at fault time with outstanding device operations.
+ Synchronization is an orthogonal issue to sharing the backing storage of a
+ buffer and hence should not be handled by dma-buf itself. This is explictly
+ mentioned here because many people seem to want something like this, but if
+ different exporters handle this differently, buffer sharing can fail in
+ interesting ways depending upong the exporter (if userspace starts depending
+ upon this implicit synchronization).
+
Miscellaneous notes
-------------------
the exporting driver to create a dmabuf fd must provide a way to let
userspace control setting of O_CLOEXEC flag passed in to dma_buf_fd().
+- If an exporter needs to manually flush caches and hence needs to fake
+ coherency for mmap support, it needs to be able to zap all the ptes pointing
+ at the backing storage. Now linux mm needs a struct address_space associated
+ with the struct file stored in vma->vm_file to do that with the function
+ unmap_mapping_range. But the dma_buf framework only backs every dma_buf fd
+ with the anon_file struct file, i.e. all dma_bufs share the same file.
+
+ Hence exporters need to setup their own file (and address_space) association
+ by setting vma->vm_file and adjusting vma->vm_pgoff in the dma_buf mmap
+ callback. In the specific case of a gem driver the exporter could use the
+ shmem file already provided by gem (and set vm_pgoff = 0). Exporters can then
+ zap ptes by unmapping the corresponding range of the struct address_space
+ associated with their own file.
+
References:
[1] struct dma_buf_ops in include/linux/dma-buf.h
[2] All interfaces mentioned above defined in include/linux/dma-buf.h
Also note the kernel might malfunction if you disable
some critical bits.
+ cma=nn[MG] [ARM,KNL]
+ Sets the size of kernel global memory area for contiguous
+ memory allocations. For more information, see
+ include/linux/dma-contiguous.h
+
cmo_free_hint= [PPC] Format: { yes | no }
Specify whether pages are marked as being inactive
when they are freed. This is used in CMO environments
a hypervisor.
Default: yes
+ coherent_pool=nn[KMG] [ARM,KNL]
+ Sets the size of memory pool for coherent, atomic dma
+ allocations.
+
code_bytes [X86] How many bytes of object code to print
in an oops report.
Range: 0 - 8192
You should also set these fields:
- v4l2_dev: set to the v4l2_device parent device.
+
- name: set to something descriptive and unique.
+
- fops: set to the v4l2_file_operations struct.
+
- ioctl_ops: if you use the v4l2_ioctl_ops to simplify ioctl maintenance
(highly recommended to use this and it might become compulsory in the
future!), then set this to your v4l2_ioctl_ops struct.
+
- lock: leave to NULL if you want to do all the locking in the driver.
Otherwise you give it a pointer to a struct mutex_lock and before any
of the v4l2_file_operations is called this lock will be taken by the
- core and released afterwards.
+ core and released afterwards. See the next section for more details.
+
- prio: keeps track of the priorities. Used to implement VIDIOC_G/S_PRIORITY.
If left to NULL, then it will use the struct v4l2_prio_state in v4l2_device.
If you want to have a separate priority state per (group of) device node(s),
then you can point it to your own struct v4l2_prio_state.
+
- parent: you only set this if v4l2_device was registered with NULL as
the parent device struct. This only happens in cases where one hardware
device has multiple PCI devices that all share the same v4l2_device core.
(cx8802). Since the v4l2_device cannot be associated with a particular
PCI device it is setup without a parent device. But when the struct
video_device is setup you do know which parent PCI device to use.
+
- flags: optional. Set to V4L2_FL_USE_FH_PRIO if you want to let the framework
handle the VIDIOC_G/S_PRIORITY ioctls. This requires that you use struct
v4l2_fh. Eventually this flag will disappear once all drivers use the core
--------------------------------
You can set a pointer to a mutex_lock in struct video_device. Usually this
-will be either a top-level mutex or a mutex per device node. If you want
-finer-grained locking then you have to set it to NULL and do you own locking.
+will be either a top-level mutex or a mutex per device node. By default this
+lock will be used for each file operation and ioctl, but you can disable
+locking for selected ioctls by calling:
+
+ void v4l2_dont_use_lock(struct video_device *vdev, unsigned int cmd);
+
+E.g.: v4l2_dont_use_lock(vdev, VIDIOC_DQBUF);
+
+You have to call this before you register the video_device.
+
+Particularly with USB drivers where certain commands such as setting controls
+can take a long time you may want to do your own locking for the buffer queuing
+ioctls.
+
+If you want still finer-grained locking then you have to set mutex_lock to NULL
+and do you own locking completely.
It is up to the driver developer to decide which method to use. However, if
your driver has high-latency operations (for example, changing the exposure
config HAVE_DMA_ATTRS
bool
+config HAVE_DMA_CONTIGUOUS
+ bool
+
config USE_GENERIC_SMP_HELPERS
bool
select HAVE_AOUT
select HAVE_DMA_API_DEBUG
select HAVE_IDE if PCI || ISA || PCMCIA
+ select HAVE_DMA_ATTRS
+ select HAVE_DMA_CONTIGUOUS if (CPU_V6 || CPU_V6K || CPU_V7)
+ select CMA if (CPU_V6 || CPU_V6K || CPU_V7)
select HAVE_MEMBLOCK
select RTC_LIB
select SYS_SUPPORTS_APM_EMULATION
config ARM_HAS_SG_CHAIN
bool
+config NEED_SG_DMA_LENGTH
+ bool
+
+config ARM_DMA_USE_IOMMU
+ select NEED_SG_DMA_LENGTH
+ select ARM_HAS_SG_CHAIN
+ bool
+
config HAVE_PWM
bool
select HAVE_S3C2410_I2C if I2C
select HAVE_S3C2410_WATCHDOG if WATCHDOG
select NEED_MACH_MEMORY_H
+ select USB_ARCH_HAS_XHCI
help
Support for SAMSUNG's EXYNOS SoCs (EXYNOS4/5)
Say Y here if you want the debug print routines to direct
their output to the serial port on MSM 8960 devices.
+ config DEBUG_S3C_UART3
+ depends on PLAT_SAMSUNG
+ bool "Use S3C UART 3 for low-level debug"
+ help
+ Say Y here if you want the debug print routines to direct
+ their output to UART 3. The port must have been initialised
+ by the boot-loader before use.
+
+ The uncompressor code port configuration is now handled
+ by CONFIG_S3C_LOWLEVEL_UART_PORT.
+
config DEBUG_REALVIEW_STD_PORT
bool "RealView Default UART"
depends on ARCH_REALVIEW
--- /dev/null
+/*
+ * Common device tree include for all Exynos 5250 boards based off of Daisy.
+ *
+ * Copyright (c) 2012 Google, Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+/ {
+ memory {
+ reg = <0x40000000 0x80000000>;
+ };
+
+ chosen {
+ };
+
+ aliases {
+ sysmmu2 = &sysmmu_2;
+ sysmmu3 = &sysmmu_3;
+ sysmmu4 = &sysmmu_4;
+ sysmmu27 = &sysmmu_27;
+ sysmmu28 = &sysmmu_28;
+ sysmmu23 = &sysmmu_23;
+ sysmmu24 = &sysmmu_24;
+ sysmmu25 = &sysmmu_25;
+ sysmmu26 = &sysmmu_26;
+ gsc0 = &gsc_0;
+ gsc1 = &gsc_1;
+ gsc2 = &gsc_2;
+ gsc3 = &gsc_3;
+ i2s0 = &i2s_0;
+ };
+
+ i2c@12C60000 {
+ samsung,i2c-sda-delay = <100>;
+ samsung,i2c-max-bus-freq = <66000>;
+ gpios = <&gpb3 0 2 3 0>,
+ <&gpb3 1 2 3 0>;
+
+ max77686_pmic@9 {
+ compatible = "maxim,max77686-pmic";
+ interrupt-parent = <&wakeup_eint>;
+ interrupts = <26 0>;
+ reg = <0x9>;
+
+ max77686,buck_ramp_delay = <2>; /* default */
+
+ voltage-regulators {
+ ldo11_reg: LDO11 {
+ regulator-name = "vdd_ldo11";
+ regulator-min-microvolt = <1900000>;
+ regulator-max-microvolt = <1900000>;
+ regulator-always-on;
+ };
+
+ ldo14_reg: LDO14 {
+ regulator-name = "vdd_ldo14";
+ regulator-min-microvolt = <1900000>;
+ regulator-max-microvolt = <1900000>;
+ regulator-always-on;
+ };
+
+ buck1_reg: BUCK1 {
+ regulator-name = "vdd_mif";
+ regulator-min-microvolt = <950000>;
+ regulator-max-microvolt = <1300000>;
+ regulator-always-on;
+ regulator-boot-on;
+ };
+
+ buck2_reg: BUCK2 {
+ regulator-name = "vdd_arm";
+ regulator-min-microvolt = <850000>;
+ regulator-max-microvolt = <1350000>;
+ regulator-always-on;
+ regulator-boot-on;
+ };
+
+ buck3_reg: BUCK3 {
+ regulator-name = "vdd_int";
+ regulator-min-microvolt = <900000>;
+ regulator-max-microvolt = <1200000>;
+ regulator-always-on;
+ regulator-boot-on;
+ };
+
+ buck4_reg: BUCK4 {
+ regulator-name = "vdd_g3d";
+ regulator-min-microvolt = <850000>;
+ regulator-max-microvolt = <1300000>;
+ regulator-boot-on;
+ };
+
+ buck8_reg: BUCK8 {
+ regulator-name = "vdd_ummc";
+ regulator-min-microvolt = <900000>;
+ regulator-max-microvolt = <3000000>;
+ regulator-always-on;
+ regulator-boot-on;
+ };
+
+ en32khz_ap: EN32KHZ_AP {
+ regulator-name = "en32khz_ap";
+ regulator-boot-on;
+ };
+
+ en32khz_cp: EN32KHZ_CP {
+ regulator-name = "en32khz_cp";
+ regulator-boot-on;
+ };
+
+ enp32kh: ENP32KHZ {
+ regulator-name = "enp32khz";
+ regulator-boot-on;
+ };
+ };
+ };
+ };
+
+ i2c@12C70000 {
+ samsung,i2c-sda-delay = <100>;
+ samsung,i2c-max-bus-freq = <66000>;
+ gpios = <&gpb3 2 2 3 0>,
+ <&gpb3 3 2 3 0>;
+
+ trackpad {
+ reg = <0x67>;
+ compatible = "cypress,cyapa";
+ interrupts = <10 0>;
+ interrupt-parent = <&wakeup_eint>;
+ };
+ };
+
+ i2c@12C80000 {
+ samsung,i2c-sda-delay = <100>;
+ samsung,i2c-max-bus-freq = <66000>;
+ gpios = <&gpa0 6 3 3 0>,
+ <&gpa0 7 3 3 0>;
+
+ exynos_hdcp@3a {
+ compatible = "samsung,exynos_hdcp";
+ reg = <0x3a>;
+ };
+ };
+
+ i2c@12C90000 {
+ samsung,i2c-sda-delay = <100>;
+ samsung,i2c-max-bus-freq = <66000>;
+ gpios = <&gpa1 2 3 3 0>,
+ <&gpa1 3 3 3 0>;
+
+ tpm {
+ compatible = "infineon,slb9635tt";
+ reg = <0x20>;
+ };
+ };
+
+ i2c@12CA0000 {
+ samsung,i2c-sda-delay = <100>;
+ samsung,i2c-max-bus-freq = <66000>;
+ gpios = <&gpa2 0 3 3 0>,
+ <&gpa2 1 3 3 0>;
+
+ power-regulator {
+ compatible = "ti,tps65090";
+ reg = <0x48>;
+
+ voltage-regulators {
+ VFET1 {
+ tps65090-control-reg-offset = <15>;
+ regulator-name = "vcd_led";
+ regulator-min-microvolt = <12000000>;
+ regulator-max-microvolt = <12000000>;
+ };
+ VFET2 {
+ tps65090-control-reg-offset = <16>;
+ regulator-name = "video_mid";
+ regulator-min-microvolt = <5000000>;
+ regulator-max-microvolt = <5000000>;
+ };
+ VFET3 {
+ tps65090-control-reg-offset = <17>;
+ regulator-name = "wwan_r";
+ regulator-min-microvolt = <3300000>;
+ regulator-max-microvolt = <3300000>;
+ regulator-always-on;
+ };
+ VFET4 {
+ tps65090-control-reg-offset = <18>;
+ regulator-name = "sdcard";
+ regulator-min-microvolt = <3300000>;
+ regulator-max-microvolt = <3300000>;
+ };
+ VFET5 {
+ tps65090-control-reg-offset = <19>;
+ regulator-name = "camout";
+ regulator-min-microvolt = <3300000>;
+ regulator-max-microvolt = <3300000>;
+ regulator-always-on;
+ };
+ VFET6 {
+ tps65090-control-reg-offset = <20>;
+ regulator-name = "lcd_vdd";
+ regulator-min-microvolt = <3300000>;
+ regulator-max-microvolt = <3300000>;
+ };
+ VFET7 {
+ tps65090-control-reg-offset = <21>;
+ regulator-name = "ts";
+ regulator-min-microvolt = <5000000>;
+ regulator-max-microvolt = <5000000>;
+ };
+ };
+ };
+
+ ec {
+ compatible = "google,chromeos-ec";
+ reg = <0x1e>;
+ interrupts = <14 0>;
+ interrupt-parent = <&wakeup_eint>;
+ };
+
+ // i2c4 hsic hub @0x8, eeprom @0x50, batt @0xb
+ };
+
+ i2c@12CB0000 {
+ samsung,i2c-sda-delay = <100>;
+ samsung,i2c-max-bus-freq = <66000>;
+ gpios = <&gpa2 2 3 3 0>,
+ <&gpa2 3 3 3 0>;
+
+ // i2c5 conn? ts?
+ };
+
+ i2c@12CC0000 {
+ status = "disabled";
+
+ // i2c6 is not used on any cros5250 boards
+ };
+
+ i2c@12CD0000 {
+ samsung,i2c-sda-delay = <100>;
+ samsung,i2c-max-bus-freq = <66000>;
+ gpios = <&gpb2 2 3 3 0>,
+ <&gpb2 3 3 3 0>;
+
+ light-sensor {
+ compatible = "invn,isl29018";
+ reg = <0x44>;
+ };
+
+ // i2c7
+ // mipi cam codec 0x11 gyro @0x68
+ // LCD @0x50-57 ALS @? mic-detect @0x3b
+ };
+
+ spi_0: spi@12d20000 {
+ status = "disabled";
+ };
+
+ spi_1: spi@12d30000 {
+ gpios = <&gpa2 4 2 3 0>,
+ <&gpa2 6 2 3 0>,
+ <&gpa2 7 2 3 0>;
+ samsung,spi-src-clk = <0>;
+ num-cs = <1>;
+ };
+
+ spi_2: spi@12d40000 {
+ status = "disabled";
+ };
+
+ dwmmc0@12200000 {
+ supports-highspeed;
+ card-detection-broken;
+ no-write-protect;
+ fifo-depth = <0x80>;
+ card-detect-delay = <200>;
+ samsung,dw-mshc-sdr-timing = <2 3 3>;
+ samsung,dw-mshc-ddr-timing = <1 2 3>;
+
+ slot0 {
+ bus-width = <8>;
+ cd-gpios = <&gpc0 2 2 3 3>;
+ gpios = <&gpc0 0 2 0 3>, <&gpc0 1 2 0 3>,
+ <&gpc1 0 2 3 3>, <&gpc1 1 2 3 3>,
+ <&gpc1 2 2 3 3>, <&gpc1 3 2 3 3>,
+ <&gpc0 3 2 3 3>, <&gpc0 4 2 3 3>,
+ <&gpc0 5 2 3 3>, <&gpc0 6 2 3 3>;
+ };
+ };
+
+ dwmmc1@12210000 {
+ status = "disabled";
+ };
+
+ dwmmc2@12220000 {
+ supports-highspeed;
+ card-detection-broken;
+ no-write-protect;
+ fifo-depth = <0x80>;
+ card-detect-delay = <200>;
+ samsung,dw-mshc-sdr-timing = <2 3 3>;
+ samsung,dw-mshc-ddr-timing = <1 2 3>;
+
+ slot0 {
+ bus-width = <4>;
+ cd-gpios = <&gpc3 2 2 3 3>;
+ gpios = <&gpc3 1 2 0 3>, <&gpc3 0 2 0 3>,
+ <&gpc3 3 2 3 3>, <&gpc3 4 2 3 3>,
+ <&gpc3 5 2 3 3>, <&gpc3 6 2 3 3>;
+ };
+ };
+
+ dwmmc3@12230000 {
+ supports-highspeed;
+ card-detection-broken;
+ no-write-protect;
+ fifo-depth = <0x80>;
+ card-detect-delay = <200>;
+ samsung,dw-mshc-sdr-timing = <2 3 3>;
+ samsung,dw-mshc-ddr-timing = <1 2 3>;
+
+ slot0 {
+ bus-width = <4>;
+ gpios = <&gpc4 1 2 3 3>, <&gpc4 0 2 0 3>,
+ <&gpc4 3 2 3 3>, <&gpc4 4 2 3 3>,
+ <&gpc4 5 2 3 3>, <&gpc4 6 2 3 3>;
+ };
+ };
+
+ i2c@12CE0000 {
+ samsung,i2c-sda-delay = <100>;
+ samsung,i2c-max-bus-freq = <66000>;
+ };
+
+ ehci {
+ samsung,vbus-gpio = <&gpx1 1 1 3 3>;
+ };
+
+ xhci {
+ samsung,vbus-gpio = <&gpx2 7 1 3 3>;
+ };
+
+ fixed-regulator {
+ compatible = "regulator-fixed";
+ regulator-name = "hsichub-reset-l";
+ gpio = <&gpe1 0 1 0 0>;
+ enable-active-high;
+ regulator-always-on;
+ };
+
+ // NB: nodes must be at root for regulator-fixed to probe
+ // NB: must set regulator-boot-on for enable-active-high to be used
+ // NB: set regulator-always-on to suppress complaints
+ // "incomplete constraints, leaving on"
+ wifi-en {
+ compatible = "regulator-fixed";
+ regulator-name = "wifi-en";
+ gpio = <&gpx0 1 0 0 0>;
+ enable-active-high;
+ regulator-boot-on;
+ regulator-always-on;
+ };
+ wifi-rst {
+ compatible = "regulator-fixed";
+ regulator-name = "wifi-rst-l";
+ gpio = <&gpx0 2 0 0 0>;
+ enable-active-high;
+ regulator-boot-on;
+ regulator-always-on;
+ };
+ bt-rst {
+ compatible = "regulator-fixed";
+ regulator-name = "bt-reset-l";
+ gpio = <&gpx3 2 0 0 0>;
+ enable-active-high;
+ regulator-boot-on;
+ regulator-always-on;
+ };
+
+ gpio-keys {
+ compatible = "gpio-keys";
+
+ power {
+ label = "Power";
+ gpios = <&gpx1 3 0 0 0>;
+ linux,code = <116>; /* KEY_POWER */
+ gpio-key,wakeup;
+ };
+ };
+};
i2c@138D0000 {
status = "disabled";
};
+
+ spi_0: spi@13920000 {
+ status = "disabled";
+ };
+
+ spi_1: spi@13930000 {
+ status = "disabled";
+ };
+
+ spi_2: spi@13940000 {
+ status = "disabled";
+ };
};
i2c@138D0000 {
status = "disabled";
};
+
+ spi_0: spi@13920000 {
+ status = "disabled";
+ };
+
+ spi_1: spi@13930000 {
+ status = "disabled";
+ };
+
+ spi_2: spi@13940000 {
+ gpios = <&gpc1 1 5 3 0>,
+ <&gpc1 3 5 3 0>,
+ <&gpc1 4 5 3 0>;
+
+ w25x80@0 {
+ #address-cells = <1>;
+ #size-cells = <1>;
+ compatible = "w25x80";
+ reg = <0>;
+ spi-max-frequency = <1000000>;
+
+ controller-data {
+ cs-gpio = <&gpc1 2 1 0 3>;
+ samsung,spi-feedback-delay = <0>;
+ };
+
+ partition@0 {
+ label = "U-Boot";
+ reg = <0x0 0x40000>;
+ read-only;
+ };
+
+ partition@40000 {
+ label = "Kernel";
+ reg = <0x40000 0xc0000>;
+ };
+ };
+ };
};
compatible = "samsung,exynos4210";
interrupt-parent = <&gic>;
+ aliases {
+ spi0 = &spi_0;
+ spi1 = &spi_1;
+ spi2 = &spi_2;
+ };
+
gic:interrupt-controller@10490000 {
compatible = "arm,cortex-a9-gic";
#interrupt-cells = <3>;
interrupts = <0 65 0>;
};
+ spi_0: spi@13920000 {
+ compatible = "samsung,exynos4210-spi";
+ reg = <0x13920000 0x100>;
+ interrupts = <0 66 0>;
+ tx-dma-channel = <&pdma0 7>;
+ rx-dma-channel = <&pdma0 6>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ };
+
+ spi_1: spi@13930000 {
+ compatible = "samsung,exynos4210-spi";
+ reg = <0x13930000 0x100>;
+ interrupts = <0 67 0>;
+ tx-dma-channel = <&pdma1 7>;
+ rx-dma-channel = <&pdma1 6>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ };
+
+ spi_2: spi@13940000 {
+ compatible = "samsung,exynos4210-spi";
+ reg = <0x13940000 0x100>;
+ interrupts = <0 68 0>;
+ tx-dma-channel = <&pdma0 9>;
+ rx-dma-channel = <&pdma0 8>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ };
+
amba {
#address-cells = <1>;
#size-cells = <1>;
--- /dev/null
+/*
+ * Google Daisy board device tree source
+ *
+ * Copyright (c) 2012 Google, Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+/dts-v1/;
+/include/ "exynos5250.dtsi"
+/include/ "cros5250-common.dtsi"
+
+/ {
+ model = "Google Daisy";
+ compatible = "google,daisy", "samsung,exynos5250";
+
+ sromc-bus {
+ lan9215@3,0 {
+ compatible = "smsc,lan9215", "smsc,lan9115";
+ reg = <3 0 0x20000>;
+ interrupts = <5 0>;
+ interrupt-parent = <&wakeup_eint>;
+ phy-mode = "mii";
+ smsc,irq-push-pull;
+ smsc,force-internal-phy;
+ local-mac-address = [00 80 00 23 45 67];
+ };
+ };
+
+ display-port-controller {
+ status = "disabled";
+ };
+};
bootargs = "root=/dev/ram0 rw ramdisk=8192 initrd=0x41000000,8M console=ttySAC3,115200 init=/linuxrc";
};
- lan9215@0x05000000 {
- compatible = "smsc,lan9215", "smsc,lan9115";
- reg = <0x05000000 0x20000>;
- interrupts = <5 0>;
- interrupt-parent = <&wakeup_eint>;
- phy-mode = "mii";
- smsc,irq-push-pull;
- smsc,force-internal-phy;
- local-mac-address = [00 80 00 23 45 67];
+ sromc-bus {
+ lan9215@1,0 {
+ compatible = "smsc,lan9215", "smsc,lan9115";
+ reg = <1 0 0x20000>;
+ interrupts = <5 0>;
+ interrupt-parent = <&wakeup_eint>;
+ phy-mode = "mii";
+ smsc,irq-push-pull;
+ smsc,force-internal-phy;
+ };
};
i2c@12C60000 {
#address-cells = <1>;
#size-cells = <0>;
samsung,i2c-sda-delay = <100>;
- samsung,i2c-max-bus-freq = <20000>;
+ samsung,i2c-max-bus-freq = <66000>;
gpios = <&gpb3 0 2 3 0>,
<&gpb3 1 2 3 0>;
compatible = "samsung,s524ad0xd1";
reg = <0x50>;
};
+
+ max77686_pmic@9 {
+ compatible = "maxim,max77686-pmic";
+ interrupt-parent = <&wakeup_eint>;
+ interrupts = <26 0>;
+ reg = <0x9>;
+
+ max77686,buck_ramp_delay = <2>; /* default */
+
+ voltage-regulators {
+ ldo11_reg: LDO11 {
+ regulator-name = "vdd_ldo11";
+ regulator-min-microvolt = <1900000>;
+ regulator-max-microvolt = <1900000>;
+ regulator-always-on;
+ };
+
+ ldo14_reg: LDO14 {
+ regulator-name = "vdd_ldo14";
+ regulator-min-microvolt = <1900000>;
+ regulator-max-microvolt = <1900000>;
+ regulator-always-on;
+ };
+
+ buck1_reg: BUCK1 {
+ regulator-name = "vdd_mif";
+ regulator-min-microvolt = <950000>;
+ regulator-max-microvolt = <1300000>;
+ regulator-always-on;
+ regulator-boot-on;
+ };
+
+ buck2_reg: BUCK2 {
+ regulator-name = "vdd_arm";
+ regulator-min-microvolt = <850000>;
+ regulator-max-microvolt = <1350000>;
+ regulator-always-on;
+ regulator-boot-on;
+ };
+
+ buck3_reg: BUCK3 {
+ regulator-name = "vdd_int";
+ regulator-min-microvolt = <900000>;
+ regulator-max-microvolt = <1200000>;
+ regulator-always-on;
+ regulator-boot-on;
+ };
+
+ buck4_reg: BUCK4 {
+ regulator-name = "vdd_g3d";
+ regulator-min-microvolt = <850000>;
+ regulator-max-microvolt = <1300000>;
+ regulator-boot-on;
+ };
+
+ buck8_reg: BUCK8 {
+ regulator-name = "vdd_ummc";
+ regulator-min-microvolt = <900000>;
+ regulator-max-microvolt = <3000000>;
+ regulator-always-on;
+ regulator-boot-on;
+ };
+ };
+ };
};
i2c@12C70000 {
#address-cells = <1>;
#size-cells = <0>;
samsung,i2c-sda-delay = <100>;
- samsung,i2c-max-bus-freq = <20000>;
+ samsung,i2c-max-bus-freq = <66000>;
gpios = <&gpb3 2 2 3 0>,
<&gpb3 3 2 3 0>;
};
i2c@12C80000 {
- status = "disabled";
+ #address-cells = <1>;
+ #size-cells = <0>;
+ samsung,i2c-sda-delay = <100>;
+ samsung,i2c-max-bus-freq = <66000>;
+ gpios = <&gpa0 6 3 3 0>,
+ <&gpa0 7 3 3 0>;
+
+ exynos_hdcp@3a {
+ compatible = "samsung,exynos_hdcp";
+ reg = <0x3a>;
+ };
};
i2c@12C90000 {
card-detect-delay = <200>;
samsung,dw-mshc-sdr-timing = <2 3 3>;
samsung,dw-mshc-ddr-timing = <1 2 3>;
-
+
slot0 {
bus-width = <8>;
cd-gpios = <&gpc0 2 2 3 3>;
};
dwmmc2@12220000 {
- supports-highspeed;
card-detection-broken;
no-write-protect;
fifo-depth = <0x80>;
card-detect-delay = <200>;
samsung,dw-mshc-sdr-timing = <2 3 3>;
samsung,dw-mshc-ddr-timing = <1 2 3>;
-
+
slot0 {
bus-width = <4>;
cd-gpios = <&gpc3 2 2 3 3>;
dwmmc3@12230000 {
status = "disabled";
};
+
+ i2c@12CE0000 {
+ #address-cells = <1>;
+ #size-cells = <0>;
+ samsung,i2c-sda-delay = <100>;
+ samsung,i2c-max-bus-freq = <66000>;
+ };
+
+ spi_0: spi@12d20000 {
+ status = "disabled";
+ };
+
+ spi_1: spi@12d30000 {
+ gpios = <&gpa2 4 2 3 0>,
+ <&gpa2 6 2 3 0>,
+ <&gpa2 7 2 3 0>;
+
+ w25q80bw@0 {
+ #address-cells = <1>;
+ #size-cells = <1>;
+ compatible = "w25x80";
+ reg = <0>;
+ spi-max-frequency = <1000000>;
+
+ controller-data {
+ cs-gpio = <&gpa2 5 1 0 3>;
+ samsung,spi-feedback-delay = <0>;
+ };
+
+ partition@0 {
+ label = "U-Boot";
+ reg = <0x0 0x40000>;
+ read-only;
+ };
+
+ partition@40000 {
+ label = "Kernel";
+ reg = <0x40000 0xc0000>;
+ };
+ };
+ };
+
+ spi_2: spi@12d40000 {
+ status = "disabled";
+ };
+
+ display-port-controller {
+ status = "disabled";
+ };
};
--- /dev/null
+/*
+ * Google Snow board device tree source
+ *
+ * Copyright (c) 2012 Google, Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+/dts-v1/;
+/include/ "exynos5250.dtsi"
+/include/ "cros5250-common.dtsi"
+
+/ {
+ model = "Google Snow";
+ compatible = "google,snow", "samsung,exynos5250";
+
+ mipi {
+ status = "disabled";
+ };
+};
compatible = "samsung,exynos5250";
interrupt-parent = <&gic>;
+ aliases {
+ mshc0 = &mshc_0;
+ mshc1 = &mshc_1;
+ mshc2 = &mshc_2;
+ mshc3 = &mshc_3;
+ sysmmu2 = &sysmmu_2;
+ sysmmu3 = &sysmmu_3;
+ sysmmu4 = &sysmmu_4;
+ sysmmu27 = &sysmmu_27;
+ sysmmu28 = &sysmmu_28;
+ sysmmu23 = &sysmmu_23;
+ sysmmu24 = &sysmmu_24;
+ sysmmu25 = &sysmmu_25;
+ sysmmu26 = &sysmmu_26;
+ gsc0 = &gsc_0;
+ gsc1 = &gsc_1;
+ gsc2 = &gsc_2;
+ gsc3 = &gsc_3;
+ i2s0 = &i2s_0;
+ i2c0 = &i2c_0;
+ i2c1 = &i2c_1;
+ i2c2 = &i2c_2;
+ i2c3 = &i2c_3;
+ i2c4 = &i2c_4;
+ i2c5 = &i2c_5;
+ i2c6 = &i2c_6;
+ i2c7 = &i2c_7;
+ i2c8 = &i2c_8;
+ spi0 = &spi_0;
+ spi1 = &spi_1;
+ spi2 = &spi_2;
+ };
+
gic:interrupt-controller@10481000 {
compatible = "arm,cortex-a9-gic";
#interrupt-cells = <3>;
interrupts = <0 42 0>;
};
+ mfc {
+ compatible = "samsung,s5p-mfc-v6";
+ reg = <0x11000000 0x10000>;
+ interrupts = <0 96 0>;
+ sysmmu_l = <&sysmmu_3>;
+ sysmmu_r = <&sysmmu_4>;
+ };
+
+ ohci {
+ compatible = "samsung,exynos-ohci";
+ reg = <0x12120000 0x100>;
+ interrupts = <0 71 0>;
+ };
+
+ ehci {
+ compatible = "samsung,exynos-ehci";
+ reg = <0x12110000 0x100>;
+ interrupts = <0 71 0>;
+ };
+
+ xhci {
+ compatible = "samsung,exynos-xhci";
+ reg = <0x12000000 0x10000>;
+ interrupts = <0 72 0>;
+ };
+
rtc {
compatible = "samsung,s3c6410-rtc";
reg = <0x101E0000 0x100>;
interrupts = <0 54 0>;
};
- i2c@12C60000 {
+ i2c_0: i2c@12C60000 {
compatible = "samsung,s3c2440-i2c";
reg = <0x12C60000 0x100>;
interrupts = <0 56 0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
};
- i2c@12C70000 {
+ i2c_1: i2c@12C70000 {
compatible = "samsung,s3c2440-i2c";
reg = <0x12C70000 0x100>;
interrupts = <0 57 0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
};
- i2c@12C80000 {
+ i2c_2: i2c@12C80000 {
compatible = "samsung,s3c2440-i2c";
reg = <0x12C80000 0x100>;
interrupts = <0 58 0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
};
- i2c@12C90000 {
+ i2c_3: i2c@12C90000 {
compatible = "samsung,s3c2440-i2c";
reg = <0x12C90000 0x100>;
interrupts = <0 59 0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
};
- i2c@12CA0000 {
+ i2c_4: i2c@12CA0000 {
compatible = "samsung,s3c2440-i2c";
reg = <0x12CA0000 0x100>;
interrupts = <0 60 0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
};
- i2c@12CB0000 {
+ i2c_5: i2c@12CB0000 {
compatible = "samsung,s3c2440-i2c";
reg = <0x12CB0000 0x100>;
interrupts = <0 61 0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
};
- i2c@12CC0000 {
+ i2c_6: i2c@12CC0000 {
compatible = "samsung,s3c2440-i2c";
reg = <0x12CC0000 0x100>;
interrupts = <0 62 0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
};
- i2c@12CD0000 {
+ i2c_7: i2c@12CD0000 {
compatible = "samsung,s3c2440-i2c";
reg = <0x12CD0000 0x100>;
interrupts = <0 63 0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ };
+
+ i2c_8: i2c@12CE0000 {
+ compatible = "samsung,s3c2440-hdmiphy-i2c";
+ reg = <0x12CE0000 0x1000>;
+ interrupts = <0 64 0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ };
+
+ sysmmu_2: sysmmu@0x10A60000 {
+ compatible = "samsung,s5p-sysmmu";
+ reg = <0x10A60000 0x100>;
+ interrupts = <24 5>;
+ interrupt-parent = <&combiner>;
+ };
+
+ sysmmu_3: sysmmu@0x11210000 {
+ compatible = "samsung,s5p-sysmmu";
+ reg = <0x11210000 0x100>;
+ interrupts = <8 5>;
+ interrupt-parent = <&combiner>;
+ };
+
+ sysmmu_4: sysmmu@0x11200000 {
+ compatible = "samsung,s5p-sysmmu";
+ reg = <0x11200000 0x100>;
+ interrupts = <6 2>;
+ interrupt-parent = <&combiner>;
+ };
+
+ sysmmu_27: sysmmu@0x14640000 {
+ compatible = "samsung,s5p-sysmmu";
+ reg = <0x14640000 0x100>;
+ interrupts = <3 2>;
+ interrupt-parent = <&combiner>;
+ };
+
+ sysmmu_28: sysmmu@0x14650000 {
+ compatible = "samsung,s5p-sysmmu";
+ reg = <0x14650000 0x100>;
+ interrupts = <7 4>;
+ interrupt-parent = <&combiner>;
+ };
+
+ sysmmu_23: sysmmu@0x13E80000 {
+ compatible = "samsung,s5p-sysmmu";
+ reg = <0x13E80000 0x100>;
+ interrupts = <2 0>;
+ interrupt-parent = <&combiner>;
+ };
+
+ sysmmu_24: sysmmu@0x13E90000 {
+ compatible = "samsung,s5p-sysmmu";
+ reg = <0x13E90000 0x100>;
+ interrupts = <2 2>;
+ interrupt-parent = <&combiner>;
};
- dwmmc0@12200000 {
+ sysmmu_25: sysmmu@0x13EA0000 {
+ compatible = "samsung,s5p-sysmmu";
+ reg = <0x13EA0000 0x100>;
+ interrupts = <2 4>;
+ interrupt-parent = <&combiner>;
+ };
+
+ sysmmu_26: sysmmu@0x13EB0000 {
+ compatible = "samsung,s5p-sysmmu";
+ reg = <0x13EB0000 0x100>;
+ interrupts = <2 6>;
+ interrupt-parent = <&combiner>;
+ };
+
+ mshc_0: dwmmc0@12200000 {
compatible = "synopsis,dw-mshc-exynos5250";
reg = <0x12200000 0x1000>;
interrupts = <0 75 0>;
};
- dwmmc1@12210000 {
+ mshc_1: dwmmc1@12210000 {
compatible = "synopsis,dw-mshc-exynos5250";
reg = <0x12210000 0x1000>;
interrupts = <0 76 0>;
};
- dwmmc2@12220000 {
+ mshc_2: dwmmc2@12220000 {
compatible = "synopsis,dw-mshc-exynos5250";
reg = <0x12220000 0x1000>;
interrupts = <0 77 0>;
};
-
- dwmmc3@12230000 {
+
+ mshc_3: dwmmc3@12230000 {
compatible = "synopsis,dw-mshc-exynos5250";
reg = <0x12230000 0x1000>;
interrupts = <0 78 0>;
};
+ i2s_0: i2s@03830000 {
+ compatible = "samsung,i2s";
+ reg = <0x03830000 0x100>;
+ tx-dma-channel-secondary = <&pdma0 8>;
+ tx-dma-channel = <&pdma0 10>;
+ rx-dma-channel = <&pdma0 9>;
+ };
+
+ spi_0: spi@12d20000 {
+ compatible = "samsung,exynos4210-spi";
+ reg = <0x12d20000 0x100>;
+ interrupts = <0 66 0>;
+ tx-dma-channel = <&pdma0 5>;
+ rx-dma-channel = <&pdma0 4>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ };
+
+ spi_1: spi@12d30000 {
+ compatible = "samsung,exynos4210-spi";
+ reg = <0x12d30000 0x100>;
+ interrupts = <0 67 0>;
+ tx-dma-channel = <&pdma1 5>;
+ rx-dma-channel = <&pdma1 4>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ };
+
+ spi_2: spi@12d40000 {
+ compatible = "samsung,exynos4210-spi";
+ reg = <0x12d40000 0x100>;
+ interrupts = <0 68 0>;
+ tx-dma-channel = <&pdma0 7>;
+ rx-dma-channel = <&pdma0 6>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ };
+
amba {
#address-cells = <1>;
#size-cells = <1>;
#gpio-cells = <4>;
};
};
+
+ fimd {
+ compatible = "samsung,exynos5-fb";
+ interrupt-parent = <&combiner>;
+ reg = <0x14400000 0x40000>;
+ interrupts = <18 4>, <18 5>, <18 6>;
+ sysmmu = <&sysmmu_27>;
+ };
+
+ mipi {
+ compatible = "samsung,exynos5-mipi";
+ reg = <0x14500000 0x10000>;
+ interrupts = <0 82 0>;
+ };
+
+ display-port-controller {
+ compatible = "samsung,exynos5-dp";
+ reg = <0x145B0000 0x10000>;
+ interrupts = <10 3>;
+ interrupt-parent = <&combiner>;
+ };
+
+ gsc_0: gsc@0x13e00000 {
+ compatible = "samsung,exynos-gsc";
+ reg = <0x13e00000 0x1000>;
+ interrupts = <0 85 0>;
+ sysmmu = <&sysmmu_23>;
+ };
+
+ gsc_1: gsc@0x13e10000 {
+ compatible = "samsung,exynos-gsc";
+ reg = <0x13e10000 0x1000>;
+ interrupts = <0 86 0>;
+ sysmmu = <&sysmmu_24>;
+ };
+
+ gsc_2: gsc@0x13e20000 {
+ compatible = "samsung,exynos-gsc";
+ reg = <0x13e20000 0x1000>;
+ interrupts = <0 87 0>;
+ sysmmu = <&sysmmu_25>;
+ };
+
+ gsc_3: gsc@0x13e30000 {
+ compatible = "samsung,exynos-gsc";
+ reg = <0x13e30000 0x1000>;
+ interrupts = <0 88 0>;
+ sysmmu = <&sysmmu_26>;
+ };
+
+ g2d {
+ compatible = "samsung,s5p-g2d";
+ reg = <0x10850000 0x400>;
+ interrupts = <0 91 0>;
+ };
+
+ hdmi {
+ compatible = "samsung,exynos5-hdmi";
+ reg = <0x14530000 0x100000>;
+ interrupts = <0 95 0>, <5 0>;
+ interrupt-parent = <&wakeup_eint>;
+ };
+
+ mixer {
+ compatible = "samsung,s5p-mixer";
+ reg = <0x14450000 0x10000>;
+ interrupts = <0 94 0>;
+ sysmmu = <&sysmmu_28>;
+ };
+
+ sromc-bus {
+ compatible = "samsung,exynos-sromc-bus", "simple-bus";
+
+ #address-cells = <2>;
+ #size-cells = <1>;
+ ranges = < 0 0 0x04000000 0x20000
+ 1 0 0x05000000 0x20000
+ 2 0 0x06000000 0x20000
+ 3 0 0x07000000 0x20000>;
+ };
};
read_lock_irqsave(&device_info->lock, flags);
list_for_each_entry(b, &device_info->safe_buffers, node)
- if (b->safe_dma_addr == safe_dma_addr) {
+ if (b->safe_dma_addr <= safe_dma_addr &&
+ b->safe_dma_addr + b->size > safe_dma_addr) {
rb = b;
break;
}
if (buf == NULL) {
dev_err(dev, "%s: unable to map unsafe buffer %p!\n",
__func__, ptr);
- return ~0;
+ return DMA_ERROR_CODE;
}
dev_dbg(dev, "%s: unsafe buffer %p (dma=%#x) mapped to %p (dma=%#x)\n",
* substitute the safe buffer for the unsafe one.
* (basically move the buffer from an unsafe area to a safe one)
*/
-dma_addr_t __dma_map_page(struct device *dev, struct page *page,
- unsigned long offset, size_t size, enum dma_data_direction dir)
+static dma_addr_t dmabounce_map_page(struct device *dev, struct page *page,
+ unsigned long offset, size_t size, enum dma_data_direction dir,
+ struct dma_attrs *attrs)
{
dma_addr_t dma_addr;
int ret;
ret = needs_bounce(dev, dma_addr, size);
if (ret < 0)
- return ~0;
+ return DMA_ERROR_CODE;
if (ret == 0) {
- __dma_page_cpu_to_dev(page, offset, size, dir);
+ arm_dma_ops.sync_single_for_device(dev, dma_addr, size, dir);
return dma_addr;
}
if (PageHighMem(page)) {
dev_err(dev, "DMA buffer bouncing of HIGHMEM pages is not supported\n");
- return ~0;
+ return DMA_ERROR_CODE;
}
return map_single(dev, page_address(page) + offset, size, dir);
}
-EXPORT_SYMBOL(__dma_map_page);
/*
* see if a mapped address was really a "safe" buffer and if so, copy
* the safe buffer. (basically return things back to the way they
* should be)
*/
-void __dma_unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size,
- enum dma_data_direction dir)
+static void dmabounce_unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size,
+ enum dma_data_direction dir, struct dma_attrs *attrs)
{
struct safe_buffer *buf;
buf = find_safe_buffer_dev(dev, dma_addr, __func__);
if (!buf) {
- __dma_page_dev_to_cpu(pfn_to_page(dma_to_pfn(dev, dma_addr)),
- dma_addr & ~PAGE_MASK, size, dir);
+ arm_dma_ops.sync_single_for_cpu(dev, dma_addr, size, dir);
return;
}
unmap_single(dev, buf, size, dir);
}
-EXPORT_SYMBOL(__dma_unmap_page);
-int dmabounce_sync_for_cpu(struct device *dev, dma_addr_t addr,
- unsigned long off, size_t sz, enum dma_data_direction dir)
+static int __dmabounce_sync_for_cpu(struct device *dev, dma_addr_t addr,
+ size_t sz, enum dma_data_direction dir)
{
struct safe_buffer *buf;
+ unsigned long off;
- dev_dbg(dev, "%s(dma=%#x,off=%#lx,sz=%zx,dir=%x)\n",
- __func__, addr, off, sz, dir);
+ dev_dbg(dev, "%s(dma=%#x,sz=%zx,dir=%x)\n",
+ __func__, addr, sz, dir);
buf = find_safe_buffer_dev(dev, addr, __func__);
if (!buf)
return 1;
+ off = addr - buf->safe_dma_addr;
+
BUG_ON(buf->direction != dir);
- dev_dbg(dev, "%s: unsafe buffer %p (dma=%#x) mapped to %p (dma=%#x)\n",
- __func__, buf->ptr, virt_to_dma(dev, buf->ptr),
+ dev_dbg(dev, "%s: unsafe buffer %p (dma=%#x off=%#lx) mapped to %p (dma=%#x)\n",
+ __func__, buf->ptr, virt_to_dma(dev, buf->ptr), off,
buf->safe, buf->safe_dma_addr);
DO_STATS(dev->archdata.dmabounce->bounce_count++);
}
return 0;
}
-EXPORT_SYMBOL(dmabounce_sync_for_cpu);
-int dmabounce_sync_for_device(struct device *dev, dma_addr_t addr,
- unsigned long off, size_t sz, enum dma_data_direction dir)
+static void dmabounce_sync_for_cpu(struct device *dev,
+ dma_addr_t handle, size_t size, enum dma_data_direction dir)
+{
+ if (!__dmabounce_sync_for_cpu(dev, handle, size, dir))
+ return;
+
+ arm_dma_ops.sync_single_for_cpu(dev, handle, size, dir);
+}
+
+static int __dmabounce_sync_for_device(struct device *dev, dma_addr_t addr,
+ size_t sz, enum dma_data_direction dir)
{
struct safe_buffer *buf;
+ unsigned long off;
- dev_dbg(dev, "%s(dma=%#x,off=%#lx,sz=%zx,dir=%x)\n",
- __func__, addr, off, sz, dir);
+ dev_dbg(dev, "%s(dma=%#x,sz=%zx,dir=%x)\n",
+ __func__, addr, sz, dir);
buf = find_safe_buffer_dev(dev, addr, __func__);
if (!buf)
return 1;
+ off = addr - buf->safe_dma_addr;
+
BUG_ON(buf->direction != dir);
- dev_dbg(dev, "%s: unsafe buffer %p (dma=%#x) mapped to %p (dma=%#x)\n",
- __func__, buf->ptr, virt_to_dma(dev, buf->ptr),
+ dev_dbg(dev, "%s: unsafe buffer %p (dma=%#x off=%#lx) mapped to %p (dma=%#x)\n",
+ __func__, buf->ptr, virt_to_dma(dev, buf->ptr), off,
buf->safe, buf->safe_dma_addr);
DO_STATS(dev->archdata.dmabounce->bounce_count++);
}
return 0;
}
-EXPORT_SYMBOL(dmabounce_sync_for_device);
+
+static void dmabounce_sync_for_device(struct device *dev,
+ dma_addr_t handle, size_t size, enum dma_data_direction dir)
+{
+ if (!__dmabounce_sync_for_device(dev, handle, size, dir))
+ return;
+
+ arm_dma_ops.sync_single_for_device(dev, handle, size, dir);
+}
+
+static int dmabounce_set_mask(struct device *dev, u64 dma_mask)
+{
+ if (dev->archdata.dmabounce)
+ return 0;
+
+ return arm_dma_ops.set_dma_mask(dev, dma_mask);
+}
+
+static struct dma_map_ops dmabounce_ops = {
+ .alloc = arm_dma_alloc,
+ .free = arm_dma_free,
+ .mmap = arm_dma_mmap,
+ .get_sgtable = arm_dma_get_sgtable,
+ .map_page = dmabounce_map_page,
+ .unmap_page = dmabounce_unmap_page,
+ .sync_single_for_cpu = dmabounce_sync_for_cpu,
+ .sync_single_for_device = dmabounce_sync_for_device,
+ .map_sg = arm_dma_map_sg,
+ .unmap_sg = arm_dma_unmap_sg,
+ .sync_sg_for_cpu = arm_dma_sync_sg_for_cpu,
+ .sync_sg_for_device = arm_dma_sync_sg_for_device,
+ .set_dma_mask = dmabounce_set_mask,
+};
static int dmabounce_init_pool(struct dmabounce_pool *pool, struct device *dev,
const char *name, unsigned long size)
#endif
dev->archdata.dmabounce = device_info;
+ set_dma_ops(dev, &dmabounce_ops);
dev_info(dev, "dmabounce: registered device\n");
struct dmabounce_device_info *device_info = dev->archdata.dmabounce;
dev->archdata.dmabounce = NULL;
+ set_dma_ops(dev, NULL);
if (!device_info) {
dev_warn(dev,
#define ASMARM_DEVICE_H
struct dev_archdata {
+ struct dma_map_ops *dma_ops;
#ifdef CONFIG_DMABOUNCE
struct dmabounce_device_info *dmabounce;
#endif
#ifdef CONFIG_IOMMU_API
void *iommu; /* private IOMMU data */
#endif
+#ifdef CONFIG_ARM_DMA_USE_IOMMU
+ struct dma_iommu_mapping *mapping;
+#endif
};
struct omap_device;
--- /dev/null
+#ifndef ASMARM_DMA_CONTIGUOUS_H
+#define ASMARM_DMA_CONTIGUOUS_H
+
+#ifdef __KERNEL__
+#ifdef CONFIG_CMA
+
+#include <linux/types.h>
+#include <asm-generic/dma-contiguous.h>
+
+void dma_contiguous_early_fixup(phys_addr_t base, unsigned long size);
+
+#endif
+#endif
+
+#endif
--- /dev/null
+#ifndef ASMARM_DMA_IOMMU_H
+#define ASMARM_DMA_IOMMU_H
+
+#ifdef __KERNEL__
+
+#include <linux/mm_types.h>
+#include <linux/scatterlist.h>
+#include <linux/dma-debug.h>
+#include <linux/kmemcheck.h>
+
+struct dma_iommu_mapping {
+ /* iommu specific data */
+ struct iommu_domain *domain;
+
+ void *bitmap;
+ size_t bits;
+ unsigned int order;
+ dma_addr_t base;
+
+ spinlock_t lock;
+ struct kref kref;
+};
+
+struct dma_iommu_mapping *
+arm_iommu_create_mapping(struct bus_type *bus, dma_addr_t base, size_t size,
+ int order);
+
+void arm_iommu_release_mapping(struct dma_iommu_mapping *mapping);
+
+int arm_iommu_attach_device(struct device *dev,
+ struct dma_iommu_mapping *mapping);
+
+#endif /* __KERNEL__ */
+#endif
#include <linux/mm_types.h>
#include <linux/scatterlist.h>
+#include <linux/dma-attrs.h>
#include <linux/dma-debug.h>
#include <asm-generic/dma-coherent.h>
#include <asm/memory.h>
+#define DMA_ERROR_CODE (~0)
+extern struct dma_map_ops arm_dma_ops;
+
+static inline struct dma_map_ops *get_dma_ops(struct device *dev)
+{
+ if (dev && dev->archdata.dma_ops)
+ return dev->archdata.dma_ops;
+ return &arm_dma_ops;
+}
+
+static inline void set_dma_ops(struct device *dev, struct dma_map_ops *ops)
+{
+ BUG_ON(!dev);
+ dev->archdata.dma_ops = ops;
+}
+
+#include <asm-generic/dma-mapping-common.h>
+
+static inline int dma_set_mask(struct device *dev, u64 mask)
+{
+ return get_dma_ops(dev)->set_dma_mask(dev, mask);
+}
+
#ifdef __arch_page_to_dma
#error Please update to __arch_pfn_to_dma
#endif
}
#endif
-/*
- * The DMA API is built upon the notion of "buffer ownership". A buffer
- * is either exclusively owned by the CPU (and therefore may be accessed
- * by it) or exclusively owned by the DMA device. These helper functions
- * represent the transitions between these two ownership states.
- *
- * Note, however, that on later ARMs, this notion does not work due to
- * speculative prefetches. We model our approach on the assumption that
- * the CPU does do speculative prefetches, which means we clean caches
- * before transfers and delay cache invalidation until transfer completion.
- *
- * Private support functions: these are not part of the API and are
- * liable to change. Drivers must not use these.
- */
-static inline void __dma_single_cpu_to_dev(const void *kaddr, size_t size,
- enum dma_data_direction dir)
-{
- extern void ___dma_single_cpu_to_dev(const void *, size_t,
- enum dma_data_direction);
-
- if (!arch_is_coherent())
- ___dma_single_cpu_to_dev(kaddr, size, dir);
-}
-
-static inline void __dma_single_dev_to_cpu(const void *kaddr, size_t size,
- enum dma_data_direction dir)
-{
- extern void ___dma_single_dev_to_cpu(const void *, size_t,
- enum dma_data_direction);
-
- if (!arch_is_coherent())
- ___dma_single_dev_to_cpu(kaddr, size, dir);
-}
-
-static inline void __dma_page_cpu_to_dev(struct page *page, unsigned long off,
- size_t size, enum dma_data_direction dir)
-{
- extern void ___dma_page_cpu_to_dev(struct page *, unsigned long,
- size_t, enum dma_data_direction);
-
- if (!arch_is_coherent())
- ___dma_page_cpu_to_dev(page, off, size, dir);
-}
-
-static inline void __dma_page_dev_to_cpu(struct page *page, unsigned long off,
- size_t size, enum dma_data_direction dir)
-{
- extern void ___dma_page_dev_to_cpu(struct page *, unsigned long,
- size_t, enum dma_data_direction);
-
- if (!arch_is_coherent())
- ___dma_page_dev_to_cpu(page, off, size, dir);
-}
-
-extern int dma_supported(struct device *, u64);
-extern int dma_set_mask(struct device *, u64);
-
/*
* DMA errors are defined by all-bits-set in the DMA address.
*/
static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
{
- return dma_addr == ~0;
+ return dma_addr == DMA_ERROR_CODE;
}
/*
{
}
+extern int dma_supported(struct device *dev, u64 mask);
+
/**
- * dma_alloc_coherent - allocate consistent memory for DMA
+ * arm_dma_alloc - allocate consistent memory for DMA
* @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
* @size: required memory size
* @handle: bus-specific DMA address
+ * @attrs: optinal attributes that specific mapping properties
*
- * Allocate some uncached, unbuffered memory for a device for
- * performing DMA. This function allocates pages, and will
- * return the CPU-viewed address, and sets @handle to be the
- * device-viewed address.
+ * Allocate some memory for a device for performing DMA. This function
+ * allocates pages, and will return the CPU-viewed address, and sets @handle
+ * to be the device-viewed address.
*/
-extern void *dma_alloc_coherent(struct device *, size_t, dma_addr_t *, gfp_t);
+extern void *arm_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle,
+ gfp_t gfp, struct dma_attrs *attrs);
+
+#define dma_alloc_coherent(d, s, h, f) dma_alloc_attrs(d, s, h, f, NULL)
+
+static inline void *dma_alloc_attrs(struct device *dev, size_t size,
+ dma_addr_t *dma_handle, gfp_t flag,
+ struct dma_attrs *attrs)
+{
+ struct dma_map_ops *ops = get_dma_ops(dev);
+ void *cpu_addr;
+ BUG_ON(!ops);
+
+ cpu_addr = ops->alloc(dev, size, dma_handle, flag, attrs);
+ debug_dma_alloc_coherent(dev, size, *dma_handle, cpu_addr);
+ return cpu_addr;
+}
/**
- * dma_free_coherent - free memory allocated by dma_alloc_coherent
+ * arm_dma_free - free memory allocated by arm_dma_alloc
* @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
* @size: size of memory originally requested in dma_alloc_coherent
* @cpu_addr: CPU-view address returned from dma_alloc_coherent
* @handle: device-view address returned from dma_alloc_coherent
+ * @attrs: optinal attributes that specific mapping properties
*
* Free (and unmap) a DMA buffer previously allocated by
- * dma_alloc_coherent().
+ * arm_dma_alloc().
*
* References to memory and mappings associated with cpu_addr/handle
* during and after this call executing are illegal.
*/
-extern void dma_free_coherent(struct device *, size_t, void *, dma_addr_t);
+extern void arm_dma_free(struct device *dev, size_t size, void *cpu_addr,
+ dma_addr_t handle, struct dma_attrs *attrs);
+
+#define dma_free_coherent(d, s, c, h) dma_free_attrs(d, s, c, h, NULL)
+
+static inline void dma_free_attrs(struct device *dev, size_t size,
+ void *cpu_addr, dma_addr_t dma_handle,
+ struct dma_attrs *attrs)
+{
+ struct dma_map_ops *ops = get_dma_ops(dev);
+ BUG_ON(!ops);
+
+ debug_dma_free_coherent(dev, size, cpu_addr, dma_handle);
+ ops->free(dev, size, cpu_addr, dma_handle, attrs);
+}
/**
- * dma_mmap_coherent - map a coherent DMA allocation into user space
+ * arm_dma_mmap - map a coherent DMA allocation into user space
* @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
* @vma: vm_area_struct describing requested user mapping
* @cpu_addr: kernel CPU-view address returned from dma_alloc_coherent
* @handle: device-view address returned from dma_alloc_coherent
* @size: size of memory originally requested in dma_alloc_coherent
+ * @attrs: optinal attributes that specific mapping properties
*
* Map a coherent DMA buffer previously allocated by dma_alloc_coherent
* into user space. The coherent DMA buffer must not be freed by the
* driver until the user space mapping has been released.
*/
-int dma_mmap_coherent(struct device *, struct vm_area_struct *,
- void *, dma_addr_t, size_t);
+extern int arm_dma_mmap(struct device *dev, struct vm_area_struct *vma,
+ void *cpu_addr, dma_addr_t dma_addr, size_t size,
+ struct dma_attrs *attrs);
+#define dma_mmap_coherent(d, v, c, h, s) dma_mmap_attrs(d, v, c, h, s, NULL)
-/**
- * dma_alloc_writecombine - allocate writecombining memory for DMA
- * @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
- * @size: required memory size
- * @handle: bus-specific DMA address
- *
- * Allocate some uncached, buffered memory for a device for
- * performing DMA. This function allocates pages, and will
- * return the CPU-viewed address, and sets @handle to be the
- * device-viewed address.
- */
-extern void *dma_alloc_writecombine(struct device *, size_t, dma_addr_t *,
- gfp_t);
+static inline int dma_mmap_attrs(struct device *dev, struct vm_area_struct *vma,
+ void *cpu_addr, dma_addr_t dma_addr,
+ size_t size, struct dma_attrs *attrs)
+{
+ struct dma_map_ops *ops = get_dma_ops(dev);
+ BUG_ON(!ops);
+ return ops->mmap(dev, vma, cpu_addr, dma_addr, size, attrs);
+}
+
+static inline void *dma_alloc_writecombine(struct device *dev, size_t size,
+ dma_addr_t *dma_handle, gfp_t flag)
+{
+ DEFINE_DMA_ATTRS(attrs);
+ dma_set_attr(DMA_ATTR_WRITE_COMBINE, &attrs);
+ return dma_alloc_attrs(dev, size, dma_handle, flag, &attrs);
+}
-#define dma_free_writecombine(dev,size,cpu_addr,handle) \
- dma_free_coherent(dev,size,cpu_addr,handle)
+static inline void dma_free_writecombine(struct device *dev, size_t size,
+ void *cpu_addr, dma_addr_t dma_handle)
+{
+ DEFINE_DMA_ATTRS(attrs);
+ dma_set_attr(DMA_ATTR_WRITE_COMBINE, &attrs);
+ return dma_free_attrs(dev, size, cpu_addr, dma_handle, &attrs);
+}
-int dma_mmap_writecombine(struct device *, struct vm_area_struct *,
- void *, dma_addr_t, size_t);
+static inline int dma_mmap_writecombine(struct device *dev, struct vm_area_struct *vma,
+ void *cpu_addr, dma_addr_t dma_addr, size_t size)
+{
+ DEFINE_DMA_ATTRS(attrs);
+ dma_set_attr(DMA_ATTR_WRITE_COMBINE, &attrs);
+ return dma_mmap_attrs(dev, vma, cpu_addr, dma_addr, size, &attrs);
+}
/*
* This can be called during boot to increase the size of the consistent
* DMA region above it's default value of 2MB. It must be called before the
* memory allocator is initialised, i.e. before any core_initcall.
*/
-extern void __init init_consistent_dma_size(unsigned long size);
-
+static inline void init_consistent_dma_size(unsigned long size) { }
-#ifdef CONFIG_DMABOUNCE
/*
* For SA-1111, IXP425, and ADI systems the dma-mapping functions are "magic"
* and utilize bounce buffers as needed to work around limited DMA windows.
*/
extern void dmabounce_unregister_dev(struct device *);
-/*
- * The DMA API, implemented by dmabounce.c. See below for descriptions.
- */
-extern dma_addr_t __dma_map_page(struct device *, struct page *,
- unsigned long, size_t, enum dma_data_direction);
-extern void __dma_unmap_page(struct device *, dma_addr_t, size_t,
- enum dma_data_direction);
-
-/*
- * Private functions
- */
-int dmabounce_sync_for_cpu(struct device *, dma_addr_t, unsigned long,
- size_t, enum dma_data_direction);
-int dmabounce_sync_for_device(struct device *, dma_addr_t, unsigned long,
- size_t, enum dma_data_direction);
-#else
-static inline int dmabounce_sync_for_cpu(struct device *d, dma_addr_t addr,
- unsigned long offset, size_t size, enum dma_data_direction dir)
-{
- return 1;
-}
-static inline int dmabounce_sync_for_device(struct device *d, dma_addr_t addr,
- unsigned long offset, size_t size, enum dma_data_direction dir)
-{
- return 1;
-}
-
-
-static inline dma_addr_t __dma_map_page(struct device *dev, struct page *page,
- unsigned long offset, size_t size, enum dma_data_direction dir)
-{
- __dma_page_cpu_to_dev(page, offset, size, dir);
- return pfn_to_dma(dev, page_to_pfn(page)) + offset;
-}
-
-static inline void __dma_unmap_page(struct device *dev, dma_addr_t handle,
- size_t size, enum dma_data_direction dir)
-{
- __dma_page_dev_to_cpu(pfn_to_page(dma_to_pfn(dev, handle)),
- handle & ~PAGE_MASK, size, dir);
-}
-#endif /* CONFIG_DMABOUNCE */
-
-/**
- * dma_map_single - map a single buffer for streaming DMA
- * @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
- * @cpu_addr: CPU direct mapped address of buffer
- * @size: size of buffer to map
- * @dir: DMA transfer direction
- *
- * Ensure that any data held in the cache is appropriately discarded
- * or written back.
- *
- * The device owns this memory once this call has completed. The CPU
- * can regain ownership by calling dma_unmap_single() or
- * dma_sync_single_for_cpu().
- */
-static inline dma_addr_t dma_map_single(struct device *dev, void *cpu_addr,
- size_t size, enum dma_data_direction dir)
-{
- unsigned long offset;
- struct page *page;
- dma_addr_t addr;
-
- BUG_ON(!virt_addr_valid(cpu_addr));
- BUG_ON(!virt_addr_valid(cpu_addr + size - 1));
- BUG_ON(!valid_dma_direction(dir));
-
- page = virt_to_page(cpu_addr);
- offset = (unsigned long)cpu_addr & ~PAGE_MASK;
- addr = __dma_map_page(dev, page, offset, size, dir);
- debug_dma_map_page(dev, page, offset, size, dir, addr, true);
-
- return addr;
-}
-
-/**
- * dma_map_page - map a portion of a page for streaming DMA
- * @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
- * @page: page that buffer resides in
- * @offset: offset into page for start of buffer
- * @size: size of buffer to map
- * @dir: DMA transfer direction
- *
- * Ensure that any data held in the cache is appropriately discarded
- * or written back.
- *
- * The device owns this memory once this call has completed. The CPU
- * can regain ownership by calling dma_unmap_page().
- */
-static inline dma_addr_t dma_map_page(struct device *dev, struct page *page,
- unsigned long offset, size_t size, enum dma_data_direction dir)
-{
- dma_addr_t addr;
-
- BUG_ON(!valid_dma_direction(dir));
-
- addr = __dma_map_page(dev, page, offset, size, dir);
- debug_dma_map_page(dev, page, offset, size, dir, addr, false);
-
- return addr;
-}
-
-/**
- * dma_unmap_single - unmap a single buffer previously mapped
- * @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
- * @handle: DMA address of buffer
- * @size: size of buffer (same as passed to dma_map_single)
- * @dir: DMA transfer direction (same as passed to dma_map_single)
- *
- * Unmap a single streaming mode DMA translation. The handle and size
- * must match what was provided in the previous dma_map_single() call.
- * All other usages are undefined.
- *
- * After this call, reads by the CPU to the buffer are guaranteed to see
- * whatever the device wrote there.
- */
-static inline void dma_unmap_single(struct device *dev, dma_addr_t handle,
- size_t size, enum dma_data_direction dir)
-{
- debug_dma_unmap_page(dev, handle, size, dir, true);
- __dma_unmap_page(dev, handle, size, dir);
-}
-
-/**
- * dma_unmap_page - unmap a buffer previously mapped through dma_map_page()
- * @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
- * @handle: DMA address of buffer
- * @size: size of buffer (same as passed to dma_map_page)
- * @dir: DMA transfer direction (same as passed to dma_map_page)
- *
- * Unmap a page streaming mode DMA translation. The handle and size
- * must match what was provided in the previous dma_map_page() call.
- * All other usages are undefined.
- *
- * After this call, reads by the CPU to the buffer are guaranteed to see
- * whatever the device wrote there.
- */
-static inline void dma_unmap_page(struct device *dev, dma_addr_t handle,
- size_t size, enum dma_data_direction dir)
-{
- debug_dma_unmap_page(dev, handle, size, dir, false);
- __dma_unmap_page(dev, handle, size, dir);
-}
-
-/**
- * dma_sync_single_range_for_cpu
- * @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
- * @handle: DMA address of buffer
- * @offset: offset of region to start sync
- * @size: size of region to sync
- * @dir: DMA transfer direction (same as passed to dma_map_single)
- *
- * Make physical memory consistent for a single streaming mode DMA
- * translation after a transfer.
- *
- * If you perform a dma_map_single() but wish to interrogate the
- * buffer using the cpu, yet do not wish to teardown the PCI dma
- * mapping, you must call this function before doing so. At the
- * next point you give the PCI dma address back to the card, you
- * must first the perform a dma_sync_for_device, and then the
- * device again owns the buffer.
- */
-static inline void dma_sync_single_range_for_cpu(struct device *dev,
- dma_addr_t handle, unsigned long offset, size_t size,
- enum dma_data_direction dir)
-{
- BUG_ON(!valid_dma_direction(dir));
-
- debug_dma_sync_single_for_cpu(dev, handle + offset, size, dir);
-
- if (!dmabounce_sync_for_cpu(dev, handle, offset, size, dir))
- return;
-
- __dma_single_dev_to_cpu(dma_to_virt(dev, handle) + offset, size, dir);
-}
-
-static inline void dma_sync_single_range_for_device(struct device *dev,
- dma_addr_t handle, unsigned long offset, size_t size,
- enum dma_data_direction dir)
-{
- BUG_ON(!valid_dma_direction(dir));
-
- debug_dma_sync_single_for_device(dev, handle + offset, size, dir);
-
- if (!dmabounce_sync_for_device(dev, handle, offset, size, dir))
- return;
-
- __dma_single_cpu_to_dev(dma_to_virt(dev, handle) + offset, size, dir);
-}
-
-static inline void dma_sync_single_for_cpu(struct device *dev,
- dma_addr_t handle, size_t size, enum dma_data_direction dir)
-{
- dma_sync_single_range_for_cpu(dev, handle, 0, size, dir);
-}
-
-static inline void dma_sync_single_for_device(struct device *dev,
- dma_addr_t handle, size_t size, enum dma_data_direction dir)
-{
- dma_sync_single_range_for_device(dev, handle, 0, size, dir);
-}
/*
* The scatter list versions of the above methods.
*/
-extern int dma_map_sg(struct device *, struct scatterlist *, int,
- enum dma_data_direction);
-extern void dma_unmap_sg(struct device *, struct scatterlist *, int,
+extern int arm_dma_map_sg(struct device *, struct scatterlist *, int,
+ enum dma_data_direction, struct dma_attrs *attrs);
+extern void arm_dma_unmap_sg(struct device *, struct scatterlist *, int,
+ enum dma_data_direction, struct dma_attrs *attrs);
+extern void arm_dma_sync_sg_for_cpu(struct device *, struct scatterlist *, int,
enum dma_data_direction);
-extern void dma_sync_sg_for_cpu(struct device *, struct scatterlist *, int,
+extern void arm_dma_sync_sg_for_device(struct device *, struct scatterlist *, int,
enum dma_data_direction);
-extern void dma_sync_sg_for_device(struct device *, struct scatterlist *, int,
- enum dma_data_direction);
-
+extern int arm_dma_get_sgtable(struct device *dev, struct sg_table *sgt,
+ void *cpu_addr, dma_addr_t dma_addr, size_t size,
+ struct dma_attrs *attrs);
#endif /* __KERNEL__ */
#endif
#define MT_MEMORY_DTCM 12
#define MT_MEMORY_ITCM 13
#define MT_MEMORY_SO 14
+#define MT_MEMORY_DMA_READY 15
#ifdef CONFIG_MMU
extern void iotable_init(struct map_desc *, int);
extern void paging_init(struct machine_desc *desc);
extern void sanity_check_meminfo(void);
extern void reboot_setup(char *str);
+extern void setup_dma_zone(struct machine_desc *desc);
unsigned int processor_id;
EXPORT_SYMBOL(processor_id);
machine_desc = mdesc;
machine_name = mdesc->name;
-#ifdef CONFIG_ZONE_DMA
- if (mdesc->dma_zone_size) {
- extern unsigned long arm_dma_zone_size;
- arm_dma_zone_size = mdesc->dma_zone_size;
- }
-#endif
+ setup_dma_zone(mdesc);
+
if (mdesc->restart_mode)
reboot_setup(&mdesc->restart_mode);
help
Common setup code for FIMD0.
-config EXYNOS4_DEV_SYSMMU
+config EXYNOS4_SETUP_DP
bool
help
- Common setup code for SYSTEM MMU in EXYNOS4
+ Common setup code for DP.
+
+config EXYNOS4_SETUP_FIMD
+ bool
+ help
+ Common setup code for FIMD.
+
+config EXYNOS_DEV_SYSMMU
+ bool
+ help
+ Common setup code for SYSTEM MMU in EXYNOS
config EXYNOS4_DEV_DWMCI
bool
help
Common setup code for SPI GPIO configurations.
+config EXYNOS4_SETUP_MIPI_DSIM
+ bool
+ depends on FB_MIPI_DSIM
+ help
+ Common setup code for MIPI_DSIM to support mainline style fimd.
+
+config EXYNOS4_SETUP_TVOUT
+ bool
+ help
+ Common setup code for TVOUT
+
# machine support
if ARCH_EXYNOS4
select SOC_EXYNOS5250
select USE_OF
select ARM_AMBA
+ select EXYNOS_DEV_SYSMMU
+ select EXYNOS4_SETUP_MIPI_DSIM
+ select EXYNOS4_SETUP_DP
+ select EXYNOS4_SETUP_USB_PHY
+ select SAMSUNG_DEV_BACKLIGHT
+ select SAMSUNG_DEV_PWM
+ select EXYNOS4_SETUP_TVOUT
help
Machine support for Samsung Exynos4 machine with device tree enabled.
Select this if a fdt blob is available for the EXYNOS4 SoC based board.
obj-$(CONFIG_HOTPLUG_CPU) += hotplug.o
+obj-$(CONFIG_ARCH_EXYNOS) += clock-audss.o
+
# machine support
obj-$(CONFIG_MACH_SMDKC210) += mach-smdkv310.o
obj-y += dev-uart.o
obj-$(CONFIG_ARCH_EXYNOS4) += dev-audio.o
obj-$(CONFIG_EXYNOS4_DEV_AHCI) += dev-ahci.o
-obj-$(CONFIG_EXYNOS4_DEV_SYSMMU) += dev-sysmmu.o
+obj-$(CONFIG_EXYNOS_DEV_SYSMMU) += dev-sysmmu.o
obj-$(CONFIG_EXYNOS4_DEV_DWMCI) += dev-dwmci.o
obj-$(CONFIG_EXYNOS_DEV_DMA) += dma.o
obj-$(CONFIG_EXYNOS4_DEV_USB_OHCI) += dev-ohci.o
obj-$(CONFIG_ARCH_EXYNOS) += setup-i2c0.o
obj-$(CONFIG_EXYNOS4_SETUP_FIMC) += setup-fimc.o
obj-$(CONFIG_EXYNOS4_SETUP_FIMD0) += setup-fimd0.o
+obj-$(CONFIG_EXYNOS4_SETUP_DP) += setup-dp.o
+obj-$(CONFIG_EXYNOS4_SETUP_FIMD) += setup-fimd.o
+obj-$(CONFIG_EXYNOS4_SETUP_MIPI_DSIM) += setup-mipidsim.o
obj-$(CONFIG_EXYNOS4_SETUP_I2C1) += setup-i2c1.o
obj-$(CONFIG_EXYNOS4_SETUP_I2C2) += setup-i2c2.o
obj-$(CONFIG_EXYNOS4_SETUP_I2C3) += setup-i2c3.o
obj-$(CONFIG_EXYNOS4_SETUP_SDHCI_GPIO) += setup-sdhci-gpio.o
obj-$(CONFIG_EXYNOS4_SETUP_USB_PHY) += setup-usb-phy.o
obj-$(CONFIG_EXYNOS4_SETUP_SPI) += setup-spi.o
+obj-$(CONFIG_EXYNOS4_SETUP_TVOUT) += setup-tvout.o
zreladdr-y += 0x40008000
params_phys-y := 0x40000100
+
+
+dtb-$(CONFIG_MACH_EXYNOS4_DT) += exynos4210-origen.dtb exynos4210-smdkv310.dtb
+dtb-$(CONFIG_MACH_EXYNOS5_DT) += exynos5250-smdk5250.dtb exynos5250-daisy.dtb exynos5250-snow.dtb
--- /dev/null
+/*
+ * Copyright (c) 2012 Samsung Electronics Co., Ltd.
+ * http://www.samsung.com
+ *
+ * Clock support for EXYNOS Audio Subsystem
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#include <linux/kernel.h>
+#include <linux/err.h>
+
+#include <plat/clock.h>
+#include <plat/s5p-clock.h>
+#include <plat/clock-clksrc.h>
+
+#include <mach/map.h>
+#include <mach/regs-audss.h>
+
+static int exynos_clk_audss_ctrl(struct clk *clk, int enable)
+{
+ return s5p_gatectrl(EXYNOS_CLKGATE_AUDSS, clk, enable);
+}
+
+static struct clk *exynos_clkset_mout_audss_list[] = {
+ &clk_ext_xtal_mux,
+ &clk_fout_epll,
+};
+
+static struct clksrc_sources clkset_mout_audss = {
+ .sources = exynos_clkset_mout_audss_list,
+ .nr_sources = ARRAY_SIZE(exynos_clkset_mout_audss_list),
+};
+
+static struct clksrc_clk exynos_clk_mout_audss = {
+ .clk = {
+ .name = "mout_audss",
+ },
+ .sources = &clkset_mout_audss,
+ .reg_src = { .reg = EXYNOS_CLKSRC_AUDSS, .shift = 0, .size = 1 },
+};
+
+static struct clksrc_clk exynos_clk_dout_audss_srp = {
+ .clk = {
+ .name = "dout_srp",
+ .parent = &exynos_clk_mout_audss.clk,
+ },
+ .reg_div = { .reg = EXYNOS_CLKDIV_AUDSS, .shift = 0, .size = 4 },
+};
+
+static struct clksrc_clk exynos_clk_dout_audss_bus = {
+ .clk = {
+ .name = "dout_bus",
+ .parent = &exynos_clk_dout_audss_srp.clk,
+ },
+ .reg_div = { .reg = EXYNOS_CLKDIV_AUDSS, .shift = 4, .size = 4 },
+};
+
+static struct clksrc_clk exynos_clk_dout_audss_i2s = {
+ .clk = {
+ .name = "dout_i2s",
+ .parent = &exynos_clk_mout_audss.clk,
+ },
+ .reg_div = { .reg = EXYNOS_CLKDIV_AUDSS, .shift = 8, .size = 4 },
+};
+
+/* Clock initialization code */
+static struct clksrc_clk *exynos_audss_clks[] = {
+ &exynos_clk_mout_audss,
+ &exynos_clk_dout_audss_srp,
+ &exynos_clk_dout_audss_bus,
+ &exynos_clk_dout_audss_i2s,
+};
+
+static struct clk exynos_init_audss_clocks[] = {
+ {
+ .name = "srpclk",
+ .parent = &exynos_clk_dout_audss_srp.clk,
+ .enable = exynos_clk_audss_ctrl,
+ .ctrlbit = EXYNOS_AUDSS_CLKGATE_RP
+ | EXYNOS_AUDSS_CLKGATE_UART
+ | EXYNOS_AUDSS_CLKGATE_TIMER,
+ }, {
+ .name = "iis",
+ .devname = "samsung-i2s.0",
+ .enable = exynos_clk_audss_ctrl,
+ .ctrlbit = EXYNOS_AUDSS_CLKGATE_I2SSPECIAL
+ | EXYNOS_AUDSS_CLKGATE_I2SBUS,
+ }, {
+ .name = "iis",
+ .devname = "samsung-i2s.4",
+ .enable = exynos_clk_audss_ctrl,
+ .ctrlbit = EXYNOS_AUDSS_CLKGATE_I2SSPECIAL
+ | EXYNOS_AUDSS_CLKGATE_I2SBUS,
+ }, {
+ .name = "pcm",
+ .devname = "samsung-pcm.0",
+ .enable = exynos_clk_audss_ctrl,
+ .ctrlbit = EXYNOS_AUDSS_CLKGATE_I2SSPECIAL
+ | EXYNOS_AUDSS_CLKGATE_I2SBUS,
+ }, {
+ .name = "pcm",
+ .devname = "samsung-pcm.4",
+ .enable = exynos_clk_audss_ctrl,
+ .ctrlbit = EXYNOS_AUDSS_CLKGATE_I2SSPECIAL
+ | EXYNOS_AUDSS_CLKGATE_I2SBUS,
+ },
+};
+
+void __init exynos_register_audss_clocks(void)
+{
+ int ptr;
+
+ for (ptr = 0; ptr < ARRAY_SIZE(exynos_audss_clks); ptr++)
+ s3c_register_clksrc(exynos_audss_clks[ptr], 1);
+
+ s3c_register_clocks(exynos_init_audss_clocks,
+ ARRAY_SIZE(exynos_init_audss_clocks));
+ s3c_disable_clocks(exynos_init_audss_clocks,
+ ARRAY_SIZE(exynos_init_audss_clocks));
+}
.ctrlbit = (1 << 13),
}, {
.name = "spi",
- .devname = "s3c64xx-spi.0",
+ .devname = "exynos4210-spi.0",
.enable = exynos4_clk_ip_peril_ctrl,
.ctrlbit = (1 << 16),
}, {
.name = "spi",
- .devname = "s3c64xx-spi.1",
+ .devname = "exynos4210-spi.1",
.enable = exynos4_clk_ip_peril_ctrl,
.ctrlbit = (1 << 17),
}, {
.name = "spi",
- .devname = "s3c64xx-spi.2",
+ .devname = "exynos4210-spi.2",
.enable = exynos4_clk_ip_peril_ctrl,
.ctrlbit = (1 << 18),
}, {
.reg_div = { .reg = EXYNOS4_CLKDIV_FSYS2, .shift = 24, .size = 8 },
};
+static struct clksrc_clk exynos4_clk_mdout_spi0 = {
+ .clk = {
+ .name = "sclk_spi_mdout",
+ .devname = "exynos4210-spi.0",
+ },
+ .sources = &exynos4_clkset_group,
+ .reg_src = { .reg = EXYNOS4_CLKSRC_PERIL1, .shift = 16, .size = 4 },
+ .reg_div = { .reg = EXYNOS4_CLKDIV_PERIL1, .shift = 0, .size = 4 },
+
+};
+
static struct clksrc_clk exynos4_clk_sclk_spi0 = {
.clk = {
.name = "sclk_spi",
- .devname = "s3c64xx-spi.0",
+ .devname = "exynos4210-spi.0",
+ .parent = &exynos4_clk_mdout_spi0.clk,
.enable = exynos4_clksrc_mask_peril1_ctrl,
.ctrlbit = (1 << 16),
},
+ .reg_div = { .reg = EXYNOS4_CLKDIV_PERIL1, .shift = 8, .size = 8 },
+};
+
+static struct clksrc_clk exynos4_clk_mdout_spi1 = {
+ .clk = {
+ .name = "sclk_spi_mdout",
+ .devname = "exynos4210-spi.1",
+ },
.sources = &exynos4_clkset_group,
- .reg_src = { .reg = EXYNOS4_CLKSRC_PERIL1, .shift = 16, .size = 4 },
- .reg_div = { .reg = EXYNOS4_CLKDIV_PERIL1, .shift = 0, .size = 4 },
+ .reg_src = { .reg = EXYNOS4_CLKSRC_PERIL1, .shift = 20, .size = 4 },
+ .reg_div = { .reg = EXYNOS4_CLKDIV_PERIL1, .shift = 16, .size = 4 },
+
};
static struct clksrc_clk exynos4_clk_sclk_spi1 = {
.clk = {
.name = "sclk_spi",
- .devname = "s3c64xx-spi.1",
+ .devname = "exynos4210-spi.1",
+ .parent = &exynos4_clk_mdout_spi1.clk,
.enable = exynos4_clksrc_mask_peril1_ctrl,
.ctrlbit = (1 << 20),
},
+ .reg_div = { .reg = EXYNOS4_CLKDIV_PERIL1, .shift = 24, .size = 8 },
+};
+
+static struct clksrc_clk exynos4_clk_mdout_spi2 = {
+ .clk = {
+ .name = "sclk_spi_mdout",
+ .devname = "exynos4210-spi.2",
+ },
.sources = &exynos4_clkset_group,
- .reg_src = { .reg = EXYNOS4_CLKSRC_PERIL1, .shift = 20, .size = 4 },
- .reg_div = { .reg = EXYNOS4_CLKDIV_PERIL1, .shift = 16, .size = 4 },
+ .reg_src = { .reg = EXYNOS4_CLKSRC_PERIL1, .shift = 24, .size = 4 },
+ .reg_div = { .reg = EXYNOS4_CLKDIV_PERIL2, .shift = 0, .size = 4 },
+
};
static struct clksrc_clk exynos4_clk_sclk_spi2 = {
.clk = {
.name = "sclk_spi",
- .devname = "s3c64xx-spi.2",
+ .devname = "exynos4210-spi.2",
+ .parent = &exynos4_clk_mdout_spi2.clk,
.enable = exynos4_clksrc_mask_peril1_ctrl,
.ctrlbit = (1 << 24),
},
- .sources = &exynos4_clkset_group,
- .reg_src = { .reg = EXYNOS4_CLKSRC_PERIL1, .shift = 24, .size = 4 },
- .reg_div = { .reg = EXYNOS4_CLKDIV_PERIL2, .shift = 0, .size = 4 },
+ .reg_div = { .reg = EXYNOS4_CLKDIV_PERIL2, .shift = 8, .size = 8 },
};
/* Clock initialization code */
&exynos4_clk_sclk_spi0,
&exynos4_clk_sclk_spi1,
&exynos4_clk_sclk_spi2,
-
+ &exynos4_clk_mdout_spi0,
+ &exynos4_clk_mdout_spi1,
+ &exynos4_clk_mdout_spi2,
};
static struct clk_lookup exynos4_clk_lookup[] = {
CLKDEV_INIT("dma-pl330.0", "apb_pclk", &exynos4_clk_pdma0),
CLKDEV_INIT("dma-pl330.1", "apb_pclk", &exynos4_clk_pdma1),
CLKDEV_INIT("dma-pl330.2", "apb_pclk", &exynos4_clk_mdma1),
- CLKDEV_INIT("s3c64xx-spi.0", "spi_busclk0", &exynos4_clk_sclk_spi0.clk),
- CLKDEV_INIT("s3c64xx-spi.1", "spi_busclk0", &exynos4_clk_sclk_spi1.clk),
- CLKDEV_INIT("s3c64xx-spi.2", "spi_busclk0", &exynos4_clk_sclk_spi2.clk),
+ CLKDEV_INIT("exynos4210-spi.0", "spi_busclk0", &exynos4_clk_sclk_spi0.clk),
+ CLKDEV_INIT("exynos4210-spi.1", "spi_busclk0", &exynos4_clk_sclk_spi1.clk),
+ CLKDEV_INIT("exynos4210-spi.2", "spi_busclk0", &exynos4_clk_sclk_spi2.clk),
};
static int xtal_rate;
return s5p_gatectrl(EXYNOS5_CLKSRC_MASK_FSYS, clk, enable);
}
+static int exynos5_clk_ip_gscl_ctrl(struct clk *clk, int enable)
+{
+ return s5p_gatectrl(EXYNOS5_CLKGATE_IP_GSCL, clk, enable);
+}
+
static int exynos5_clksrc_mask_gscl_ctrl(struct clk *clk, int enable)
{
return s5p_gatectrl(EXYNOS5_CLKSRC_MASK_GSCL, clk, enable);
return s5p_gatectrl(EXYNOS5_CLKGATE_IP_PERIS, clk, enable);
}
+static int exynos5_clksrc_mask_peric1_ctrl(struct clk *clk, int enable)
+{
+ return s5p_gatectrl(EXYNOS5_CLKSRC_MASK_PERIC1, clk, enable);
+}
+
+static int exynos5_clk_ip_acp_ctrl(struct clk *clk, int enable)
+{
+ return s5p_gatectrl(EXYNOS5_CLKGATE_IP_ACP, clk, enable);
+}
+
+static int exynos5_clk_ip_isp0_ctrl(struct clk *clk, int enable)
+{
+ return s5p_gatectrl(EXYNOS5_CLKGATE_IP_ISP0, clk, enable);
+}
+
+static int exynos5_clk_ip_isp1_ctrl(struct clk *clk, int enable)
+{
+ return s5p_gatectrl(EXYNOS5_CLKGATE_IP_ISP1, clk, enable);
+}
+
+static int exynos5_clk_hdmiphy_ctrl(struct clk *clk, int enable)
+{
+ return s5p_gatectrl(S5P_HDMI_PHY_CONTROL, clk, enable);
+}
+
/* Core list of CMU_CPU side */
static struct clksrc_clk exynos5_clk_mout_apll = {
.ctrlbit = (1 << 25),
}, {
.name = "mfc",
- .devname = "s5p-mfc",
+ .devname = "s5p-mfc-v6",
.enable = exynos5_clk_ip_mfc_ctrl,
.ctrlbit = (1 << 0),
}, {
.name = "hdmi",
- .devname = "exynos4-hdmi",
+ .devname = "exynos5-hdmi",
.enable = exynos5_clk_ip_disp1_ctrl,
.ctrlbit = (1 << 6),
+ }, {
+ .name = "hdmiphy",
+ .devname = "exynos5-hdmi",
+ .enable = exynos5_clk_hdmiphy_ctrl,
+ .ctrlbit = (1 << 0),
}, {
.name = "mixer",
.devname = "s5p-mixer",
.name = "dsim0",
.enable = exynos5_clk_ip_disp1_ctrl,
.ctrlbit = (1 << 3),
+ }, {
+ .name = "fimd",
+ .devname = "exynos5-fb",
+ .enable = exynos5_clk_ip_disp1_ctrl,
+ .ctrlbit = (1 << 0),
}, {
.name = "iis",
.devname = "samsung-i2s.1",
.name = "usbhost",
.enable = exynos5_clk_ip_fsys_ctrl ,
.ctrlbit = (1 << 18),
+ }, {
+ .name = "usbdrd30",
+ .parent = &exynos5_clk_aclk_200.clk,
+ .enable = exynos5_clk_ip_fsys_ctrl,
+ .ctrlbit = (1 << 19),
}, {
.name = "usbotg",
.enable = exynos5_clk_ip_fsys_ctrl,
.ctrlbit = (1 << 13),
}, {
.name = "i2c",
- .devname = "s3c2440-hdmiphy-i2c",
.parent = &exynos5_clk_aclk_66.clk,
.enable = exynos5_clk_ip_peric_ctrl,
.ctrlbit = (1 << 14),
+ }, {
+ .name = "gscl",
+ .devname = "exynos-gsc.0",
+ .enable = exynos5_clk_ip_gscl_ctrl,
+ .ctrlbit = (1 << 0),
+ }, {
+ .name = "gscl",
+ .devname = "exynos-gsc.1",
+ .enable = exynos5_clk_ip_gscl_ctrl,
+ .ctrlbit = (1 << 1),
+ }, {
+ .name = "gscl",
+ .devname = "exynos-gsc.2",
+ .enable = exynos5_clk_ip_gscl_ctrl,
+ .ctrlbit = (1 << 2),
+ }, {
+ .name = "gscl",
+ .devname = "exynos-gsc.3",
+ .enable = exynos5_clk_ip_gscl_ctrl,
+ .ctrlbit = (1 << 3),
+ }, {
+ .name = "fimg2d",
+ .enable = exynos5_clk_ip_acp_ctrl,
+ .ctrlbit = (1 << 3),
+ }, {
+ .name = "dp",
+ .devname = "s5p-dp",
+ .enable = exynos5_clk_ip_disp1_ctrl,
+ .ctrlbit = (1 << 4),
+ }, {
+ .name = SYSMMU_CLOCK_NAME,
+ .devname = SYSMMU_CLOCK_DEVNAME(mfc_l, 3),
+ .enable = &exynos5_clk_ip_mfc_ctrl,
+ /* There is change in the MFC_L & MFC_R in v6.5 */
+ .ctrlbit = (1 << 2),
+ .parent = &exynos5_init_clocks_off[10],
+ }, {
+ .name = SYSMMU_CLOCK_NAME,
+ .devname = SYSMMU_CLOCK_DEVNAME(mfc_r, 4),
+ .enable = &exynos5_clk_ip_mfc_ctrl,
+ .ctrlbit = (1 << 1),
+ .parent = &exynos5_init_clocks_off[10],
+ }, {
+ .name = SYSMMU_CLOCK_NAME,
+ .devname = SYSMMU_CLOCK_DEVNAME(tv, 28),
+ .enable = &exynos5_clk_ip_disp1_ctrl,
+ .ctrlbit = (1 << 9),
+ .parent = &exynos5_init_clocks_off[13],
+ }, {
+ .name = SYSMMU_CLOCK_NAME,
+ .devname = SYSMMU_CLOCK_DEVNAME(jpeg, 7),
+ .enable = &exynos5_clk_ip_gen_ctrl,
+ .ctrlbit = (1 << 7),
+ }, {
+ .name = SYSMMU_CLOCK_NAME,
+ .devname = SYSMMU_CLOCK_DEVNAME(rot, 5),
+ .enable = &exynos5_clk_ip_gen_ctrl,
+ .ctrlbit = (1 << 6)
+ }, {
+ .name = SYSMMU_CLOCK_NAME,
+ .devname = SYSMMU_CLOCK_DEVNAME(gsc0, 23),
+ .enable = &exynos5_clk_ip_gscl_ctrl,
+ .ctrlbit = (1 << 7),
+ .parent = &exynos5_init_clocks_off[40],
+ }, {
+ .name = SYSMMU_CLOCK_NAME,
+ .devname = SYSMMU_CLOCK_DEVNAME(gsc1, 24),
+ .enable = &exynos5_clk_ip_gscl_ctrl,
+ .ctrlbit = (1 << 8),
+ .parent = &exynos5_init_clocks_off[41],
+ }, {
+ .name = SYSMMU_CLOCK_NAME,
+ .devname = SYSMMU_CLOCK_DEVNAME(gsc2, 25),
+ .enable = &exynos5_clk_ip_gscl_ctrl,
+ .ctrlbit = (1 << 9),
+ .parent = &exynos5_init_clocks_off[42],
+ }, {
+ .name = SYSMMU_CLOCK_NAME,
+ .devname = SYSMMU_CLOCK_DEVNAME(gsc3, 26),
+ .enable = &exynos5_clk_ip_gscl_ctrl,
+ .ctrlbit = (1 << 10),
+ .parent = &exynos5_init_clocks_off[43],
+ }, {
+ .name = SYSMMU_CLOCK_NAME,
+ .devname = SYSMMU_CLOCK_DEVNAME(isp, 11),
+ .enable = &exynos5_clk_ip_isp0_ctrl,
+ .ctrlbit = (0x3F << 8),
+ }, {
+ .name = SYSMMU_CLOCK_NAME2,
+ .devname = SYSMMU_CLOCK_DEVNAME(isp, 11),
+ .enable = &exynos5_clk_ip_isp1_ctrl,
+ .ctrlbit = (0xF << 4),
+ }, {
+ .name = SYSMMU_CLOCK_NAME,
+ .devname = SYSMMU_CLOCK_DEVNAME(2d, 2),
+ .enable = &exynos5_clk_ip_acp_ctrl,
+ .ctrlbit = (1 << 7),
}
};
.nr_sources = ARRAY_SIZE(exynos5_clkset_group_list),
};
+struct clk *exynos5_clkset_usbdrd30_list[] = {
+ [0] = &exynos5_clk_mout_mpll.clk,
+ [1] = &exynos5_clk_mout_cpll.clk,
+};
+
+struct clksrc_sources exynos5_clkset_usbdrd30 = {
+ .sources = exynos5_clkset_usbdrd30_list,
+ .nr_sources = ARRAY_SIZE(exynos5_clkset_usbdrd30_list),
+};
+
/* Possible clock sources for aclk_266_gscl_sub Mux */
static struct clk *clk_src_gscl_266_list[] = {
[0] = &clk_ext_xtal_mux,
{
.clk = {
.name = "sclk_fimd",
- .devname = "s3cfb.1",
+ .devname = "exynos5-fb",
.enable = exynos5_clksrc_mask_disp1_0_ctrl,
.ctrlbit = (1 << 0),
},
.parent = &exynos5_clk_mout_cpll.clk,
},
.reg_div = { .reg = EXYNOS5_CLKDIV_GEN, .shift = 4, .size = 3 },
+ }, {
+ .clk = {
+ .name = "sclk_usbdrd30",
+ .enable = exynos5_clksrc_mask_fsys_ctrl,
+ .ctrlbit = (1 << 28),
+ },
+ .sources = &exynos5_clkset_usbdrd30,
+ .reg_src = { .reg = EXYNOS5_CLKSRC_FSYS, .shift = 28, .size = 1 },
+ .reg_div = { .reg = EXYNOS5_CLKDIV_FSYS0, .shift = 24, .size = 4 },
+ },
+};
+
+/* For ACLK_300_gscl_mid */
+static struct clksrc_clk exynos5_clk_mout_aclk_300_gscl_mid = {
+ .clk = {
+ .name = "mout_aclk_300_gscl_mid",
+ },
+ .sources = &exynos5_clkset_aclk,
+ .reg_src = { .reg = EXYNOS5_CLKSRC_TOP0, .shift = 24, .size = 1 },
+};
+
+/* For ACLK_300_gscl */
+struct clk *exynos5_clkset_aclk_300_gscl_list[] = {
+ [0] = &exynos5_clk_mout_aclk_300_gscl_mid.clk,
+ [1] = &exynos5_clk_sclk_vpll.clk,
+};
+
+struct clksrc_sources exynos5_clkset_aclk_300_gscl = {
+ .sources = exynos5_clkset_aclk_300_gscl_list,
+ .nr_sources = ARRAY_SIZE(exynos5_clkset_aclk_300_gscl_list),
+};
+
+static struct clksrc_clk exynos5_clk_mout_aclk_300_gscl = {
+ .clk = {
+ .name = "mout_aclk_300_gscl",
+ },
+ .sources = &exynos5_clkset_aclk_300_gscl,
+ .reg_src = { .reg = EXYNOS5_CLKSRC_TOP0, .shift = 25, .size = 1 },
+};
+
+static struct clksrc_clk exynos5_clk_dout_aclk_300_gscl = {
+ .clk = {
+ .name = "dout_aclk_300_gscl",
+ .parent = &exynos5_clk_mout_aclk_300_gscl.clk,
+ },
+ .reg_div = { .reg = EXYNOS5_CLKDIV_TOP1, .shift = 12, .size = 3 },
+};
+
+/* Possible clock sources for aclk_300_gscl_sub Mux */
+static struct clk *clk_src_gscl_300_list[] = {
+ [0] = &clk_ext_xtal_mux,
+ [1] = &exynos5_clk_dout_aclk_300_gscl.clk,
+};
+
+static struct clksrc_sources clk_src_gscl_300 = {
+ .sources = clk_src_gscl_300_list,
+ .nr_sources = ARRAY_SIZE(clk_src_gscl_300_list),
+};
+
+static struct clksrc_clk exynos5_clk_aclk_300_gscl = {
+ .clk = {
+ .name = "aclk_300_gscl",
},
+ .sources = &clk_src_gscl_300,
+ .reg_src = { .reg = EXYNOS5_CLKSRC_TOP3, .shift = 10, .size = 1 },
};
/* Clock initialization code */
&exynos5_clk_aclk_266,
&exynos5_clk_aclk_200,
&exynos5_clk_aclk_166,
+ &exynos5_clk_mout_aclk_300_gscl_mid,
+ &exynos5_clk_mout_aclk_300_gscl,
+ &exynos5_clk_dout_aclk_300_gscl,
+ &exynos5_clk_aclk_300_gscl,
&exynos5_clk_aclk_66_pre,
&exynos5_clk_aclk_66,
&exynos5_clk_dout_mmc0,
.pfn = __phys_to_pfn(EXYNOS4_PA_HSPHY),
.length = SZ_4K,
.type = MT_DEVICE,
+ }, {
+ .virtual = (unsigned long)S5P_VA_AUDSS,
+ .pfn = __phys_to_pfn(EXYNOS_PA_AUDSS),
+ .length = SZ_4K,
+ .type = MT_DEVICE,
},
};
.pfn = __phys_to_pfn(EXYNOS5_PA_GIC_DIST),
.length = SZ_4K,
.type = MT_DEVICE,
+ }, {
+ .virtual = (unsigned long)S3C_VA_USB_HSPHY,
+ .pfn = __phys_to_pfn(EXYNOS5_PA_USB_PHY),
+ .length = SZ_256K,
+ .type = MT_DEVICE,
+ }, {
+ .virtual = (unsigned long)S5P_VA_DRD_PHY,
+ .pfn = __phys_to_pfn(EXYNOS5_PA_DRD_PHY),
+ .length = SZ_256K,
+ .type = MT_DEVICE,
+ }, {
+ .virtual = (unsigned long)S5P_VA_AUDSS,
+ .pfn = __phys_to_pfn(EXYNOS_PA_AUDSS),
+ .length = SZ_4K,
+ .type = MT_DEVICE,
},
};
__raw_writel(0x1, EXYNOS_SWRESET);
}
+static void wdt_reset_init(void)
+{
+ unsigned int value;
+
+ value = __raw_readl(EXYNOS5_AUTOMATIC_WDT_RESET_DISABLE);
+ value &= ~EXYNOS5_SYS_WDTRESET;
+ __raw_writel(value, EXYNOS5_AUTOMATIC_WDT_RESET_DISABLE);
+
+ value = __raw_readl(EXYNOS5_MASK_WDT_RESET_REQUEST);
+ value &= ~EXYNOS5_SYS_WDTRESET;
+ __raw_writel(value, EXYNOS5_MASK_WDT_RESET_REQUEST);
+}
+
/*
* exynos_map_io
*
s5p_init_cpu(S5P_VA_CHIPID);
s3c_init_cpu(samsung_cpu_id, cpu_ids, ARRAY_SIZE(cpu_ids));
+
+ /* TO support Watch dog reset */
+ wdt_reset_init();
}
static void __init exynos4_map_io(void)
exynos4_register_clocks();
exynos4_setup_clocks();
+ exynos_register_audss_clocks();
}
static void __init exynos5_init_clocks(int xtal)
exynos5_register_clocks();
exynos5_setup_clocks();
+ exynos_register_audss_clocks();
}
#define COMBINER_ENABLE_SET 0x0
extern struct sys_timer exynos4_timer;
void exynos_init_io(struct map_desc *mach_desc, int size);
+void exynos_register_audss_clocks(void);
void exynos4_init_irq(void);
void exynos5_init_irq(void);
void exynos4_restart(char mode, const char *cmd);
-/* linux/arch/arm/mach-exynos4/dev-sysmmu.c
+/* linux/arch/arm/mach-exynos/dev-sysmmu.c
*
* Copyright (c) 2010 Samsung Electronics Co., Ltd.
* http://www.samsung.com
*
- * EXYNOS4 - System MMU support
+ * EXYNOS - System MMU support
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
#include <linux/platform_device.h>
#include <linux/dma-mapping.h>
-#include <linux/export.h>
-
+#include <linux/of_platform.h>
#include <mach/map.h>
#include <mach/irqs.h>
#include <mach/sysmmu.h>
#include <plat/s5p-clock.h>
+#include <asm/dma-iommu.h>
+#include <linux/slab.h>
-/* These names must be equal to the clock names in mach-exynos4/clock.c */
-const char *sysmmu_ips_name[EXYNOS4_SYSMMU_TOTAL_IPNUM] = {
- "SYSMMU_MDMA" ,
- "SYSMMU_SSS" ,
- "SYSMMU_FIMC0" ,
- "SYSMMU_FIMC1" ,
- "SYSMMU_FIMC2" ,
- "SYSMMU_FIMC3" ,
- "SYSMMU_JPEG" ,
- "SYSMMU_FIMD0" ,
- "SYSMMU_FIMD1" ,
- "SYSMMU_PCIe" ,
- "SYSMMU_G2D" ,
- "SYSMMU_ROTATOR",
- "SYSMMU_MDMA2" ,
- "SYSMMU_TV" ,
- "SYSMMU_MFC_L" ,
- "SYSMMU_MFC_R" ,
-};
-
-static struct resource exynos4_sysmmu_resource[] = {
- [0] = {
- .start = EXYNOS4_PA_SYSMMU_MDMA,
- .end = EXYNOS4_PA_SYSMMU_MDMA + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [1] = {
- .start = IRQ_SYSMMU_MDMA0_0,
- .end = IRQ_SYSMMU_MDMA0_0,
- .flags = IORESOURCE_IRQ,
- },
- [2] = {
- .start = EXYNOS4_PA_SYSMMU_SSS,
- .end = EXYNOS4_PA_SYSMMU_SSS + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [3] = {
- .start = IRQ_SYSMMU_SSS_0,
- .end = IRQ_SYSMMU_SSS_0,
- .flags = IORESOURCE_IRQ,
- },
- [4] = {
- .start = EXYNOS4_PA_SYSMMU_FIMC0,
- .end = EXYNOS4_PA_SYSMMU_FIMC0 + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [5] = {
- .start = IRQ_SYSMMU_FIMC0_0,
- .end = IRQ_SYSMMU_FIMC0_0,
- .flags = IORESOURCE_IRQ,
- },
- [6] = {
- .start = EXYNOS4_PA_SYSMMU_FIMC1,
- .end = EXYNOS4_PA_SYSMMU_FIMC1 + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [7] = {
- .start = IRQ_SYSMMU_FIMC1_0,
- .end = IRQ_SYSMMU_FIMC1_0,
- .flags = IORESOURCE_IRQ,
- },
- [8] = {
- .start = EXYNOS4_PA_SYSMMU_FIMC2,
- .end = EXYNOS4_PA_SYSMMU_FIMC2 + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [9] = {
- .start = IRQ_SYSMMU_FIMC2_0,
- .end = IRQ_SYSMMU_FIMC2_0,
- .flags = IORESOURCE_IRQ,
- },
- [10] = {
- .start = EXYNOS4_PA_SYSMMU_FIMC3,
- .end = EXYNOS4_PA_SYSMMU_FIMC3 + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [11] = {
- .start = IRQ_SYSMMU_FIMC3_0,
- .end = IRQ_SYSMMU_FIMC3_0,
- .flags = IORESOURCE_IRQ,
- },
- [12] = {
- .start = EXYNOS4_PA_SYSMMU_JPEG,
- .end = EXYNOS4_PA_SYSMMU_JPEG + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [13] = {
- .start = IRQ_SYSMMU_JPEG_0,
- .end = IRQ_SYSMMU_JPEG_0,
- .flags = IORESOURCE_IRQ,
- },
- [14] = {
- .start = EXYNOS4_PA_SYSMMU_FIMD0,
- .end = EXYNOS4_PA_SYSMMU_FIMD0 + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [15] = {
- .start = IRQ_SYSMMU_LCD0_M0_0,
- .end = IRQ_SYSMMU_LCD0_M0_0,
- .flags = IORESOURCE_IRQ,
- },
- [16] = {
- .start = EXYNOS4_PA_SYSMMU_FIMD1,
- .end = EXYNOS4_PA_SYSMMU_FIMD1 + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [17] = {
- .start = IRQ_SYSMMU_LCD1_M1_0,
- .end = IRQ_SYSMMU_LCD1_M1_0,
- .flags = IORESOURCE_IRQ,
- },
- [18] = {
- .start = EXYNOS4_PA_SYSMMU_PCIe,
- .end = EXYNOS4_PA_SYSMMU_PCIe + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [19] = {
- .start = IRQ_SYSMMU_PCIE_0,
- .end = IRQ_SYSMMU_PCIE_0,
- .flags = IORESOURCE_IRQ,
- },
- [20] = {
- .start = EXYNOS4_PA_SYSMMU_G2D,
- .end = EXYNOS4_PA_SYSMMU_G2D + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [21] = {
- .start = IRQ_SYSMMU_2D_0,
- .end = IRQ_SYSMMU_2D_0,
- .flags = IORESOURCE_IRQ,
- },
- [22] = {
- .start = EXYNOS4_PA_SYSMMU_ROTATOR,
- .end = EXYNOS4_PA_SYSMMU_ROTATOR + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [23] = {
- .start = IRQ_SYSMMU_ROTATOR_0,
- .end = IRQ_SYSMMU_ROTATOR_0,
- .flags = IORESOURCE_IRQ,
- },
- [24] = {
- .start = EXYNOS4_PA_SYSMMU_MDMA2,
- .end = EXYNOS4_PA_SYSMMU_MDMA2 + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [25] = {
- .start = IRQ_SYSMMU_MDMA1_0,
- .end = IRQ_SYSMMU_MDMA1_0,
- .flags = IORESOURCE_IRQ,
- },
- [26] = {
- .start = EXYNOS4_PA_SYSMMU_TV,
- .end = EXYNOS4_PA_SYSMMU_TV + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [27] = {
- .start = IRQ_SYSMMU_TV_M0_0,
- .end = IRQ_SYSMMU_TV_M0_0,
- .flags = IORESOURCE_IRQ,
- },
- [28] = {
- .start = EXYNOS4_PA_SYSMMU_MFC_L,
- .end = EXYNOS4_PA_SYSMMU_MFC_L + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [29] = {
- .start = IRQ_SYSMMU_MFC_M0_0,
- .end = IRQ_SYSMMU_MFC_M0_0,
- .flags = IORESOURCE_IRQ,
- },
- [30] = {
- .start = EXYNOS4_PA_SYSMMU_MFC_R,
- .end = EXYNOS4_PA_SYSMMU_MFC_R + SZ_64K - 1,
- .flags = IORESOURCE_MEM,
- },
- [31] = {
- .start = IRQ_SYSMMU_MFC_M1_0,
- .end = IRQ_SYSMMU_MFC_M1_0,
- .flags = IORESOURCE_IRQ,
- },
-};
+#ifdef CONFIG_ARM_DMA_USE_IOMMU
+struct dma_iommu_mapping * s5p_create_iommu_mapping(struct device *client,
+ dma_addr_t base, unsigned int size,
+ int order, struct dma_iommu_mapping *mapping)
+{
+ if (!client)
+ return NULL;
-struct platform_device exynos4_device_sysmmu = {
- .name = "s5p-sysmmu",
- .id = 32,
- .num_resources = ARRAY_SIZE(exynos4_sysmmu_resource),
- .resource = exynos4_sysmmu_resource,
-};
-EXPORT_SYMBOL(exynos4_device_sysmmu);
+ if (mapping == NULL) {
+ mapping = arm_iommu_create_mapping(&platform_bus_type,
+ base, size, order);
+ if (!mapping)
+ return NULL;
+ }
-static struct clk *sysmmu_clk[S5P_SYSMMU_TOTAL_IPNUM];
-void sysmmu_clk_init(struct device *dev, sysmmu_ips ips)
-{
- sysmmu_clk[ips] = clk_get(dev, sysmmu_ips_name[ips]);
- if (IS_ERR(sysmmu_clk[ips]))
- sysmmu_clk[ips] = NULL;
- else
- clk_put(sysmmu_clk[ips]);
+ client->dma_parms = kzalloc(sizeof(*client->dma_parms), GFP_KERNEL);
+ dma_set_max_seg_size(client, 0xffffffffu);
+ arm_iommu_attach_device(client, mapping);
+ return mapping;
}
-void sysmmu_clk_enable(sysmmu_ips ips)
+struct platform_device *find_sysmmu_dt(struct platform_device *pdev,
+ char *name)
{
- if (sysmmu_clk[ips])
- clk_enable(sysmmu_clk[ips]);
-}
+ struct device_node *dn, *dns;
+ struct platform_device *pds;
+ const __be32 *parp;
-void sysmmu_clk_disable(sysmmu_ips ips)
-{
- if (sysmmu_clk[ips])
- clk_disable(sysmmu_clk[ips]);
+ dn = pdev->dev.of_node;
+ parp = of_get_property(dn, name, NULL);
+ if (parp==NULL) {
+ printk(KERN_ERR "Could not find property SYSMMU\n");
+ return NULL;
+ }
+ dns = of_find_node_by_phandle(be32_to_cpup(parp));
+ if (dns==NULL) {
+ printk(KERN_ERR "Could not find node\n");
+ return NULL;
+ }
+
+ pds = of_find_device_by_node(dns);
+ if (!pds) {
+ printk(KERN_ERR "No platform device found\n");
+ return NULL;
+ }
+ return pds;
}
+#endif
+
+
#define IRQ_IIC6 EXYNOS4_IRQ_IIC6
#define IRQ_IIC7 EXYNOS4_IRQ_IIC7
+#define IRQ_SPI0 EXYNOS4_IRQ_SPI0
+#define IRQ_SPI1 EXYNOS4_IRQ_SPI1
+#define IRQ_SPI2 EXYNOS4_IRQ_SPI2
+
#define IRQ_USB_HOST EXYNOS4_IRQ_USB_HOST
#define IRQ_HSMMC0 EXYNOS4_IRQ_HSMMC0
#define EXYNOS5_IRQ_DP1_INTP1 IRQ_SPI(113)
#define EXYNOS5_IRQ_CEC IRQ_SPI(114)
#define EXYNOS5_IRQ_SATA IRQ_SPI(115)
+#define EXYNOS5_GPU_IRQ_NUMBER IRQ_SPI(117)
+#define EXYNOS5_JOB_IRQ_NUMBER IRQ_SPI(118)
+#define EXYNOS5_MMU_IRQ_NUMBER IRQ_SPI(119)
#define EXYNOS5_IRQ_MCT_L0 IRQ_SPI(120)
#define EXYNOS5_IRQ_MCT_L1 IRQ_SPI(121)
#define EXYNOS5_IRQ_SYSMMU_MDMA1_1 COMBINER_IRQ(7, 3)
#define EXYNOS5_IRQ_SYSMMU_TV_0 COMBINER_IRQ(7, 4)
#define EXYNOS5_IRQ_SYSMMU_TV_1 COMBINER_IRQ(7, 5)
+#define EXYNOS5_IRQ_SYSMMU_GPSX_0 COMBINER_IRQ(7, 6)
+#define EXYNOS5_IRQ_SYSMMU_GPSX_1 COMBINER_IRQ(7, 7)
#define EXYNOS5_IRQ_SYSMMU_MFC_L_0 COMBINER_IRQ(8, 5)
#define EXYNOS5_IRQ_SYSMMU_MFC_L_1 COMBINER_IRQ(8, 6)
#define EXYNOS4_PA_G2D 0x12800000
+#define EXYNOS_PA_AUDSS 0x03810000
#define EXYNOS4_PA_I2S0 0x03830000
#define EXYNOS4_PA_I2S1 0xE3100000
#define EXYNOS4_PA_I2S2 0xE2A00000
#define EXYNOS4_PA_SPI0 0x13920000
#define EXYNOS4_PA_SPI1 0x13930000
#define EXYNOS4_PA_SPI2 0x13940000
+#define EXYNOS5_PA_SPI0 0x12D20000
+#define EXYNOS5_PA_SPI1 0x12D30000
+#define EXYNOS5_PA_SPI2 0x12D40000
#define EXYNOS4_PA_GPIO1 0x11400000
#define EXYNOS4_PA_GPIO2 0x11000000
#define EXYNOS4_PA_UART 0x13800000
#define EXYNOS5_PA_UART 0x12C00000
+#define EXYNOS5_PA_USB_PHY 0x12130000
+#define EXYNOS5_PA_DRD_PHY 0x12100000
+
#define EXYNOS4_PA_VP 0x12C00000
#define EXYNOS4_PA_MIXER 0x12C10000
#define EXYNOS4_PA_SDO 0x12C20000
#define EXYNOS4_PA_SDRAM 0x40000000
#define EXYNOS5_PA_SDRAM 0x40000000
+#define EXYNOS5_PA_G3D 0x11800000
+
/* Compatibiltiy Defines */
#define S3C_PA_HSMMC0 EXYNOS4_PA_HSMMC(0)
#define EXYNOS4_AUDSS_INT_MEM (0x03000000)
+#define EXYNOS_AUDSSREG(x) (S5P_VA_AUDSS + (x))
+
+#define EXYNOS_CLKSRC_AUDSS_OFFSET 0x0
+#define EXYNOS_CLKDIV_AUDSS_OFFSET 0x4
+#define EXYNOS_CLKGATE_AUDSS_OFFSET 0x8
+
+#define EXYNOS_CLKSRC_AUDSS (EXYNOS_AUDSSREG \
+ (EXYNOS_CLKSRC_AUDSS_OFFSET))
+#define EXYNOS_CLKDIV_AUDSS (EXYNOS_AUDSSREG \
+ (EXYNOS_CLKDIV_AUDSS_OFFSET))
+#define EXYNOS_CLKGATE_AUDSS (EXYNOS_AUDSSREG \
+ (EXYNOS_CLKGATE_AUDSS_OFFSET))
+
+/* IP Clock Gate 0 Registers */
+#define EXYNOS_AUDSS_CLKGATE_RP (1<<0)
+#define EXYNOS_AUDSS_CLKGATE_I2SBUS (1<<2)
+#define EXYNOS_AUDSS_CLKGATE_I2SSPECIAL (1<<3)
+#define EXYNOS_AUDSS_CLKGATE_UART (1<<7)
+#define EXYNOS_AUDSS_CLKGATE_TIMER (1<<8)
+
#endif /* _PLAT_REGS_AUDSS_H */
#define EXYNOS5_CLKSRC_CORE1 EXYNOS_CLKREG(0x04204)
#define EXYNOS5_CLKGATE_IP_CORE EXYNOS_CLKREG(0x04900)
+#define EXYNOS5_CLKGATE_ISP0 EXYNOS_CLKREG(0x0C800)
+
#define EXYNOS5_CLKDIV_ACP EXYNOS_CLKREG(0x08500)
#define EXYNOS5_CLKSRC_MASK_DISP1_0 EXYNOS_CLKREG(0x1032C)
#define EXYNOS5_CLKSRC_MASK_FSYS EXYNOS_CLKREG(0x10340)
#define EXYNOS5_CLKSRC_MASK_PERIC0 EXYNOS_CLKREG(0x10350)
+#define EXYNOS5_CLKSRC_MASK_PERIC1 EXYNOS_CLKREG(0x10354)
#define EXYNOS5_CLKDIV_TOP0 EXYNOS_CLKREG(0x10510)
#define EXYNOS5_CLKDIV_TOP1 EXYNOS_CLKREG(0x10514)
#define EXYNOS5_CLKDIV_PERIC0 EXYNOS_CLKREG(0x10558)
#define EXYNOS5_CLKGATE_IP_ACP EXYNOS_CLKREG(0x08800)
+#define EXYNOS5_CLKGATE_IP_ISP0 EXYNOS_CLKREG(0x0C800)
+#define EXYNOS5_CLKGATE_IP_ISP1 EXYNOS_CLKREG(0x0C804)
#define EXYNOS5_CLKGATE_IP_GSCL EXYNOS_CLKREG(0x10920)
#define EXYNOS5_CLKGATE_IP_DISP1 EXYNOS_CLKREG(0x10928)
#define EXYNOS5_CLKGATE_IP_MFC EXYNOS_CLKREG(0x1092C)
#define S5P_HDMI_PHY_CONTROL S5P_PMUREG(0x0700)
#define S5P_HDMI_PHY_ENABLE (1 << 0)
+/* only for EXYNOS5250*/
+#define S5P_USBDRD_PHY_CONTROL S5P_PMUREG(0x0704)
+#define S5P_USBDRD_PHY_ENABLE (1 << 0)
+
#define S5P_DAC_PHY_CONTROL S5P_PMUREG(0x070C)
#define S5P_DAC_PHY_ENABLE (1 << 0)
#define S5P_MIPI_DPHY_SRESETN (1 << 1)
#define S5P_MIPI_DPHY_MRESETN (1 << 2)
+#define S5P_DPTX_PHY_CONTROL S5P_PMUREG(0x720)
+#define S5P_DPTX_PHY_ENABLE (1 << 0)
+
#define S5P_INFORM0 S5P_PMUREG(0x0800)
#define S5P_INFORM1 S5P_PMUREG(0x0804)
#define S5P_INFORM2 S5P_PMUREG(0x0808)
#define S5P_SECSS_MEM_OPTION S5P_PMUREG(0x2EC8)
#define S5P_ROTATOR_MEM_OPTION S5P_PMUREG(0x2F48)
+#define EXYNOS5_SYS_DISP1BLK_CFG S5P_SYSREG(0x214)
+#define ENABLE_FIMDBYPASS_DISP1 (1 << 15)
+
#endif /* __ASM_ARCH_REGS_PMU_H */
#define EXYNOS4_HSOTG_PHYREG(x) ((x) + S3C_VA_USB_HSPHY)
+/* Exynos 4 */
#define EXYNOS4_PHYPWR EXYNOS4_HSOTG_PHYREG(0x00)
#define PHY1_HSIC_NORMAL_MASK (0xf << 9)
#define PHY1_HSIC1_SLEEP (1 << 12)
#define EXYNOS4_PHY1CON EXYNOS4_HSOTG_PHYREG(0x34)
#define FPENABLEN (1 << 0)
+
+/* Exynos 5 */
+#define EXYNOS5_PHY_HOST_CTRL0 EXYNOS4_HSOTG_PHYREG(0x00)
+#define HOST_CTRL0_PHYSWRSTALL (0x1 << 31)
+#define HOST_CTRL0_REFCLKSEL(val) (val << 19)
+#define EXYNOS5_CLKSEL_50M (0x7)
+#define EXYNOS5_CLKSEL_24M (0x5)
+#define EXYNOS5_CLKSEL_20M (0x4)
+#define EXYNOS5_CLKSEL_19200K (0x3)
+#define EXYNOS5_CLKSEL_12M (0x2)
+#define EXYNOS5_CLKSEL_10M (0x1)
+#define EXYNOS5_CLKSEL_9600K (0x0)
+#define HOST_CTRL0_CLKSEL_SHIFT (16)
+#define HOST_CTRL0_FSEL_MASK (0x7 << 16)
+
+#define HOST_CTRL0_COMMONON_N (0x1 << 9)
+#define HOST_CTRL0_SIDDQ (0x1 << 6)
+#define HOST_CTRL0_FORCESLEEP (0x1 << 5)
+#define HOST_CTRL0_FORCESUSPEND (0x1 << 4)
+#define HOST_CTRL0_WORDINTERFACE (0x1 << 3)
+#define HOST_CTRL0_UTMISWRST (0x1 << 2)
+#define HOST_CTRL0_LINKSWRST (0x1 << 1)
+#define HOST_CTRL0_PHYSWRST (0x1 << 0)
+
+#define EXYNOS5_PHY_HOST_TUNE0 EXYNOS4_HSOTG_PHYREG(0x04)
+#define EXYNOS5_PHY_HOST_TEST0 EXYNOS4_HSOTG_PHYREG(0x08)
+
+#define EXYNOS5_PHY_HSIC_CTRL1 EXYNOS4_HSOTG_PHYREG(0x10)
+#define EXYNOS5_PHY_HSIC_CTRL2 EXYNOS4_HSOTG_PHYREG(0x20)
+#define HSIC_CTRL_REFCLKSEL(val) ((val&0x3) << 23)
+#define HSIC_CTRL_REFCLKDIV(val) ((val&0x7f) << 16)
+#define HSIC_CTRL_SIDDQ (0x1 << 6)
+#define HSIC_CTRL_FORCESLEEP (0x1 << 5)
+#define HSIC_CTRL_FORCESUSPEND (0x1 << 4)
+#define HSIC_CTRL_WORDINTERFACE (0x1 << 3)
+#define HSIC_CTRL_UTMISWRST (0x1 << 2)
+#define HSIC_CTRL_PHYSWRST (0x1 << 0)
+
+#define EXYNOS5_PHY_HOST_EHCICTRL EXYNOS4_HSOTG_PHYREG(0x30)
+#define EHCICTRL_ENAINCRXALIGN (0x1 << 29)
+#define EHCICTRL_ENAINCR4 (0x1 << 28)
+#define EHCICTRL_ENAINCR8 (0x1 << 27)
+#define EHCICTRL_ENAINCR16 (0x1 << 26)
+
+#define EXYNOS5_PHY_HOST_OHCICTRL EXYNOS4_HSOTG_PHYREG(0x34)
+
+#define EXYNOS5_PHY_OTG_SYS EXYNOS4_HSOTG_PHYREG(0x38)
+#define OTG_SYS_PHYLINK_SW_RESET (0x1 << 14)
+#define OTG_SYS_LINK_SW_RST_UOTG (0x1 << 13)
+#define OTG_SYS_PHY0_SW_RST (0x1 << 12)
+#define OTG_SYS_REF_CLK_SEL(val) ((val&0x3) << 9)
+#define OTG_SYS_REF_CLK_SEL_MASK (0x3 << 9)
+#define OTG_SYS_IP_PULLUP_UOTG (0x1 << 8)
+#define OTG_SYS_COMMON_ON (0x1 << 7)
+#define OTG_SYS_CLKSEL_SHIFT (4)
+#define OTG_SYS_CTRL0_FSEL_MASK (0x7 << 4)
+#define OTG_SYS_FORCE_SLEEP (0x1 << 3)
+#define OTG_SYS_OTGDISABLE (0x1 << 2)
+#define OTG_SYS_SIDDQ_UOTG (0x1 << 1)
+#define OTG_SYS_FORCE_SUSPEND (0x1 << 0)
+
#endif /* __PLAT_S5P_REGS_USB_PHY_H */
-/* linux/arch/arm/mach-exynos4/include/mach/sysmmu.h
+/* linux/arch/arm/mach-exynos/include/mach/sysmmu.h
*
* Copyright (c) 2010-2011 Samsung Electronics Co., Ltd.
* http://www.samsung.com
#ifndef __ASM_ARM_ARCH_SYSMMU_H
#define __ASM_ARM_ARCH_SYSMMU_H __FILE__
+#include <linux/device.h>
-enum exynos4_sysmmu_ips {
- SYSMMU_MDMA,
- SYSMMU_SSS,
- SYSMMU_FIMC0,
- SYSMMU_FIMC1,
- SYSMMU_FIMC2,
- SYSMMU_FIMC3,
- SYSMMU_JPEG,
- SYSMMU_FIMD0,
- SYSMMU_FIMD1,
- SYSMMU_PCIe,
- SYSMMU_G2D,
- SYSMMU_ROTATOR,
- SYSMMU_MDMA2,
- SYSMMU_TV,
- SYSMMU_MFC_L,
- SYSMMU_MFC_R,
- EXYNOS4_SYSMMU_TOTAL_IPNUM,
-};
-
-#define S5P_SYSMMU_TOTAL_IPNUM EXYNOS4_SYSMMU_TOTAL_IPNUM
+#define SYSMMU_CLOCK_NAME "sysmmu"
+#define SYSMMU_CLOCK_NAME2 "sysmmu_mc"
+#define SYSMMU_DEVNAME_BASE "s5p-sysmmu"
+#define SYSMMU_CLOCK_DEVNAME(ipname, id) (SYSMMU_DEVNAME_BASE "." #id)
-extern const char *sysmmu_ips_name[EXYNOS4_SYSMMU_TOTAL_IPNUM];
-
-typedef enum exynos4_sysmmu_ips sysmmu_ips;
+struct sysmmu_platform_data {
+ char *dbgname;
+ char *clockname;
+};
-void sysmmu_clk_init(struct device *dev, sysmmu_ips ips);
-void sysmmu_clk_enable(sysmmu_ips ips);
-void sysmmu_clk_disable(sysmmu_ips ips);
+static inline void platform_set_sysmmu(
+ struct device *sysmmu, struct device *dev)
+{
+ dev->archdata.iommu = sysmmu;
+}
+struct dma_iommu_mapping *s5p_create_iommu_mapping(struct device *client,
+ dma_addr_t base, unsigned int size, int order,
+ struct dma_iommu_mapping *mapping);
+struct platform_device *find_sysmmu_dt(struct platform_device *pdev,
+ char *name);
#endif /* __ASM_ARM_ARCH_SYSMMU_H */
"exynos4-sdhci.3", NULL),
OF_DEV_AUXDATA("samsung,s3c2440-i2c", EXYNOS4_PA_IIC(0),
"s3c2440-i2c.0", NULL),
+ OF_DEV_AUXDATA("samsung,exynos4210-spi", EXYNOS4_PA_SPI0,
+ "exynos4210-spi.0", NULL),
+ OF_DEV_AUXDATA("samsung,exynos4210-spi", EXYNOS4_PA_SPI1,
+ "exynos4210-spi.1", NULL),
+ OF_DEV_AUXDATA("samsung,exynos4210-spi", EXYNOS4_PA_SPI2,
+ "exynos4210-spi.2", NULL),
OF_DEV_AUXDATA("arm,pl330", EXYNOS4_PA_PDMA0, "dma-pl330.0", NULL),
OF_DEV_AUXDATA("arm,pl330", EXYNOS4_PA_PDMA1, "dma-pl330.1", NULL),
{},
* published by the Free Software Foundation.
*/
+#include <linux/gpio.h>
#include <linux/of_platform.h>
+#include <linux/platform_data/dwc3-exynos.h>
+#include <linux/regulator/fixed.h>
+#include <linux/regulator/machine.h>
#include <linux/serial_core.h>
#include <linux/smsc911x.h>
+#include <linux/delay.h>
+#include <linux/i2c.h>
+#include <linux/pwm_backlight.h>
+#include <linux/mfd/wm8994/pdata.h>
+#include <linux/regulator/machine.h>
+#include <linux/spi/spi.h>
#include <asm/mach/arch.h>
#include <asm/hardware/gic.h>
#include <mach/map.h>
+#include <mach/ohci.h>
+#include <mach/regs-pmu.h>
+#include <mach/sysmmu.h>
+#include <mach/ohci.h>
+#include <mach/regs-audss.h>
+#include <plat/audio.h>
#include <plat/cpu.h>
+#include <plat/dsim.h>
+#include <plat/fb.h>
+#include <plat/mipi_dsi.h>
+#include <plat/gpio-cfg.h>
+#include <plat/regs-fb.h>
#include <plat/regs-serial.h>
#include <plat/regs-srom.h>
+#include <plat/backlight.h>
+#include <plat/devs.h>
+#include <plat/usb-phy.h>
+#include <plat/ehci.h>
+#include <plat/dp.h>
+#include <plat/s3c64xx-spi.h>
+#include <video/platform_lcd.h>
+
+#include "drm/exynos_drm.h"
#include "common.h"
static void __init smsc911x_init(int ncs)
S5P_SROM_BC0 + (ncs * 4));
}
+static struct s3c_fb_pd_win smdk5250_fb_win0 = {
+ .win_mode = {
+ .left_margin = 4,
+ .right_margin = 4,
+ .upper_margin = 4,
+ .lower_margin = 4,
+ .hsync_len = 4,
+ .vsync_len = 4,
+ .xres = 1280,
+ .yres = 800,
+ },
+ .virtual_x = 1280,
+ .virtual_y = 800 * 2,
+ .width = 223,
+ .height = 125,
+ .max_bpp = 32,
+ .default_bpp = 24,
+};
+
+static struct s3c_fb_pd_win smdk5250_fb_win1 = {
+ .win_mode = {
+ .left_margin = 4,
+ .right_margin = 4,
+ .upper_margin = 4,
+ .lower_margin = 4,
+ .hsync_len = 4,
+ .vsync_len = 4,
+ .xres = 1280,
+ .yres = 800,
+ },
+ .virtual_x = 1280,
+ .virtual_y = 800 * 2,
+ .width = 223,
+ .height = 125,
+ .max_bpp = 32,
+ .default_bpp = 24,
+};
+
+static struct s3c_fb_pd_win smdk5250_fb_win2 = {
+ .win_mode = {
+ .left_margin = 0x4,
+ .right_margin = 0x4,
+ .upper_margin = 4,
+ .lower_margin = 4,
+ .hsync_len = 4,
+ .vsync_len = 4,
+ .xres = 1280,
+ .yres = 800,
+ },
+ .virtual_x = 1280,
+ .virtual_y = 800 * 2,
+ .width = 223,
+ .height = 125,
+ .max_bpp = 32,
+ .default_bpp = 24,
+};
+
+static struct fb_videomode snow_fb_window = {
+ .left_margin = 0x80,
+ .right_margin = 0x48,
+ .upper_margin = 14,
+ .lower_margin = 3,
+ .hsync_len = 5,
+ .vsync_len = 32,
+ .xres = 1366,
+ .yres = 768,
+};
+
+static void exynos_fimd_gpio_setup_24bpp(void)
+{
+ unsigned int reg = 0;
+
+ /*
+ * Set DISP1BLK_CFG register for Display path selection
+ * FIMD of DISP1_BLK Bypass selection : DISP1BLK_CFG[15]
+ * ---------------------
+ * 0 | MIE/MDNIE
+ * 1 | FIMD : selected
+ */
+ reg = __raw_readl(S3C_VA_SYS + 0x0214);
+ reg &= ~(1 << 15); /* To save other reset values */
+ reg |= (1 << 15);
+ __raw_writel(reg, S3C_VA_SYS + 0x0214);
+}
+
+static void exynos_dp_gpio_setup_24bpp(void)
+{
+ exynos_fimd_gpio_setup_24bpp();
+
+ /* Set Hotplug detect for DP */
+ gpio_request(EXYNOS5_GPX0(7), "DP hotplug");
+ s3c_gpio_cfgpin(EXYNOS5_GPX0(7), S3C_GPIO_SFN(3));
+}
+
+#ifdef CONFIG_DRM_EXYNOS_FIMD
+static struct exynos_drm_fimd_pdata smdk5250_lcd1_pdata = {
+ .panel.timing = {
+ .xres = 1280,
+ .yres = 800,
+ .hsync_len = 4,
+ .left_margin = 0x4,
+ .right_margin = 0x4,
+ .vsync_len = 4,
+ .upper_margin = 4,
+ .lower_margin = 4,
+ .refresh = 60,
+ },
+ .vidcon0 = VIDCON0_VIDOUT_RGB | VIDCON0_PNRMODE_RGB,
+ .vidcon1 = VIDCON1_INV_VCLK,
+ .default_win = 0,
+ .bpp = 32,
+ .clock_rate = 800 * 1000 * 1000,
+};
+#else
+static struct s3c_fb_platdata smdk5250_lcd1_pdata __initdata = {
+ .win[0] = &smdk5250_fb_win0,
+ .win[1] = &smdk5250_fb_win1,
+ .win[2] = &smdk5250_fb_win2,
+ .default_win = 0,
+ .vidcon0 = VIDCON0_VIDOUT_RGB | VIDCON0_PNRMODE_RGB,
+ .vidcon1 = VIDCON1_INV_VCLK,
+ .setup_gpio = exynos_fimd_gpio_setup_24bpp,
+ .clock_rate = 800 * 1000 * 1000,
+};
+#endif
+
+static struct mipi_dsim_config dsim_info = {
+ .e_interface = DSIM_VIDEO,
+ .e_pixel_format = DSIM_24BPP_888,
+ /* main frame fifo auto flush at VSYNC pulse */
+ .auto_flush = false,
+ .eot_disable = false,
+ .auto_vertical_cnt = false,
+ .hse = false,
+ .hfp = false,
+ .hbp = false,
+ .hsa = false,
+
+ .e_no_data_lane = DSIM_DATA_LANE_4,
+ .e_byte_clk = DSIM_PLL_OUT_DIV8,
+ .e_burst_mode = DSIM_BURST,
+
+ .p = 3,
+ .m = 115,
+ .s = 1,
+
+ /* D-PHY PLL stable time spec :min = 200usec ~ max 400usec */
+ .pll_stable_time = 500,
+
+ .esc_clk = 0.4 * 1000000, /* escape clk : 10MHz */
+
+ /* stop state holding counter after bta change count 0 ~ 0xfff */
+ .stop_holding_cnt = 0x0f,
+ .bta_timeout = 0xff, /* bta timeout 0 ~ 0xff */
+ .rx_timeout = 0xffff, /* lp rx timeout 0 ~ 0xffff */
+
+ .dsim_ddi_pd = &tc358764_mipi_lcd_driver,
+};
+
+static struct mipi_dsim_lcd_config dsim_lcd_info = {
+ .rgb_timing.left_margin = 0x4,
+ .rgb_timing.right_margin = 0x4,
+ .rgb_timing.upper_margin = 0x4,
+ .rgb_timing.lower_margin = 0x4,
+ .rgb_timing.hsync_len = 0x4,
+ .rgb_timing.vsync_len = 0x4,
+ .cpu_timing.cs_setup = 0,
+ .cpu_timing.wr_setup = 1,
+ .cpu_timing.wr_act = 0,
+ .cpu_timing.wr_hold = 0,
+ .lcd_size.width = 1280,
+ .lcd_size.height = 800,
+};
+
+static struct s5p_platform_mipi_dsim dsim_platform_data = {
+ .clk_name = "dsim0",
+ .dsim_config = &dsim_info,
+ .dsim_lcd_config = &dsim_lcd_info,
+
+ .part_reset = s5p_dsim_part_reset,
+ .init_d_phy = s5p_dsim_init_d_phy,
+ .get_fb_frame_done = NULL,
+ .trigger = NULL,
+
+ /*
+ * the stable time of needing to write data on SFR
+ * when the mipi mode becomes LP mode.
+ */
+ .delay_for_stabilization = 600,
+};
+
+static struct platform_device exynos_drm_device = {
+ .name = "exynos-drm",
+ .dev = {
+ .dma_mask = &exynos_drm_device.dev.coherent_dma_mask,
+ .coherent_dma_mask = 0xffffffffUL,
+ }
+};
+
+static void lcd_set_power(struct plat_lcd_data *pd,
+ unsigned int power)
+{
+ if (of_machine_is_compatible("google,daisy") ||
+ of_machine_is_compatible("google,snow")) {
+ struct regulator *lcd_fet;
+
+ lcd_fet = regulator_get(NULL, "lcd_vdd");
+ if (!IS_ERR(lcd_fet)) {
+ if (power)
+ regulator_enable(lcd_fet);
+ else
+ regulator_disable(lcd_fet);
+
+ regulator_put(lcd_fet);
+ }
+ }
+
+ /* TODO(dianders): GPX1(5) isn't reset for snow. Fix to hold high? */
+
+
+ /* reset */
+ gpio_request_one(EXYNOS5_GPX1(5), GPIOF_OUT_INIT_HIGH, "GPX1");
+
+ mdelay(20);
+ if (power) {
+ /* fire nRESET on power up */
+ gpio_set_value(EXYNOS5_GPX1(5), 0);
+ mdelay(20);
+ gpio_set_value(EXYNOS5_GPX1(5), 1);
+ mdelay(20);
+ gpio_free(EXYNOS5_GPX1(5));
+ } else {
+ /* fire nRESET on power off */
+ gpio_set_value(EXYNOS5_GPX1(5), 0);
+ mdelay(20);
+ gpio_set_value(EXYNOS5_GPX1(5), 1);
+ mdelay(20);
+ gpio_free(EXYNOS5_GPX1(5));
+ }
+ mdelay(20);
+
+
+ /* Turn on regulator for backlight */
+ if (of_machine_is_compatible("google,daisy") ||
+ of_machine_is_compatible("google,snow")) {
+ struct regulator *backlight_fet;
+
+ backlight_fet = regulator_get(NULL, "vcd_led");
+ if (!IS_ERR(backlight_fet)) {
+ if (power)
+ regulator_enable(backlight_fet);
+ else
+ regulator_disable(backlight_fet);
+
+ regulator_put(backlight_fet);
+ }
+ /* Wait 10 ms between regulator on and PWM start per spec */
+ mdelay(10);
+ }
+
+ /*
+ * Request lcd_bl_en GPIO for smdk5250_bl_notify().
+ * TODO: Fix this so we are not at risk of requesting the GPIO
+ * multiple times, this should be done with device tree, and
+ * likely integrated into the plat-samsung/dev-backlight.c init.
+ */
+ gpio_request_one(EXYNOS5_GPX3(0), GPIOF_OUT_INIT_LOW, "GPX3");
+}
+
+static int smdk5250_match_fb(struct plat_lcd_data *pd, struct fb_info *info)
+{
+ /* Don't call .set_power callback while unblanking */
+ return 0;
+}
+
+static struct plat_lcd_data smdk5250_lcd_data = {
+ .set_power = lcd_set_power,
+ .match_fb = smdk5250_match_fb,
+};
+
+static struct platform_device smdk5250_lcd = {
+ .name = "platform-lcd",
+ .dev.platform_data = &smdk5250_lcd_data,
+};
+
+static int smdk5250_bl_notify(struct device *unused, int brightness)
+{
+ /* manage lcd_bl_en signal */
+ if (brightness)
+ gpio_set_value(EXYNOS5_GPX3(0), 1);
+ else
+ gpio_set_value(EXYNOS5_GPX3(0), 0);
+
+ return brightness;
+}
+
+/* LCD Backlight data */
+static struct samsung_bl_gpio_info smdk5250_bl_gpio_info = {
+ .no = EXYNOS5_GPB2(0),
+ .func = S3C_GPIO_SFN(2),
+};
+
+static struct platform_pwm_backlight_data smdk5250_bl_data = {
+ .pwm_period_ns = 1000000,
+ .notify = smdk5250_bl_notify,
+};
+
+struct platform_device exynos_device_md0 = {
+ .name = "exynos-mdev",
+ .id = 0,
+};
+
+struct platform_device exynos_device_md1 = {
+ .name = "exynos-mdev",
+ .id = 1,
+};
+
+struct platform_device exynos_device_md2 = {
+ .name = "exynos-mdev",
+ .id = 2,
+};
+
+static struct regulator_consumer_supply wm8994_avdd1_supply =
+ REGULATOR_SUPPLY("AVDD1", "1-001a");
+
+static struct regulator_consumer_supply wm8994_dcvdd_supply =
+ REGULATOR_SUPPLY("DCVDD", "1-001a");
+
+static struct regulator_init_data wm8994_ldo1_data = {
+ .constraints = {
+ .name = "AVDD1",
+ },
+ .num_consumer_supplies = 1,
+ .consumer_supplies = &wm8994_avdd1_supply,
+};
+
+static struct regulator_init_data wm8994_ldo2_data = {
+ .constraints = {
+ .name = "DCVDD",
+ },
+ .num_consumer_supplies = 1,
+ .consumer_supplies = &wm8994_dcvdd_supply,
+};
+
+static struct wm8994_pdata wm8994_platform_data = {
+ /* configure gpio1 function: 0x0001(Logic level input/output) */
+ .gpio_defaults[0] = 0x0001,
+ /* If the i2s0 and i2s2 is enabled simultaneously */
+ .gpio_defaults[7] = 0x8100, /* GPIO8 DACDAT3 in */
+ .gpio_defaults[8] = 0x0100, /* GPIO9 ADCDAT3 out */
+ .gpio_defaults[9] = 0x0100, /* GPIO10 LRCLK3 out */
+ .gpio_defaults[10] = 0x0100,/* GPIO11 BCLK3 out */
+ .ldo[0] = { 0, &wm8994_ldo1_data },
+ .ldo[1] = { 0, &wm8994_ldo2_data },
+};
+
+static struct i2c_board_info i2c_devs1[] __initdata = {
+ {
+ I2C_BOARD_INFO("wm8994", 0x1a),
+ .platform_data = &wm8994_platform_data,
+ },
+};
+
+static struct s3c64xx_spi_csinfo spi1_csi[] = {
+ [0] = {
+ .line = EXYNOS5_GPA2(5),
+ .fb_delay = 0x2,
+ },
+};
+
+static struct spi_board_info spi1_board_info[] __initdata = {
+ {
+ .modalias = "spidev",
+ .platform_data = NULL,
+ .max_speed_hz = 10*1000*1000,
+ .bus_num = 1,
+ .chip_select = 0,
+ .mode = SPI_MODE_0,
+ .controller_data = spi1_csi,
+ }
+};
+
+struct sysmmu_platform_data platdata_sysmmu_mfc_l = {
+ .dbgname = "mfc_l",
+ .clockname = "sysmmu",
+};
+
+struct sysmmu_platform_data platdata_sysmmu_mfc_r = {
+ .dbgname = "mfc_r",
+ .clockname = "sysmmu",
+};
+
+struct sysmmu_platform_data platdata_sysmmu_gsc = {
+ .dbgname = "gsc",
+ .clockname = "sysmmu",
+};
+
+struct sysmmu_platform_data platdata_sysmmu_g2d = {
+ .dbgname = "g2d",
+ .clockname = "sysmmu",
+};
+
+struct sysmmu_platform_data platdata_sysmmu_fimd = {
+ .dbgname = "fimd",
+ .clockname = "sysmmu",
+};
+
+struct sysmmu_platform_data platdata_sysmmu_tv = {
+ .dbgname = "tv",
+ .clockname = "sysmmu",
+};
+
+#ifdef CONFIG_VIDEO_FIMG2D4X
+static struct fimg2d_platdata fimg2d_data __initdata = {
+ .hw_ver = 0x42,
+ .gate_clkname = "fimg2d",
+};
+#endif
+
+static struct exynos4_ohci_platdata smdk5250_ohci_pdata = {
+ .phy_init = s5p_usb_phy_init,
+ .phy_exit = s5p_usb_phy_exit,
+};
+
+static struct s5p_ehci_platdata smdk5250_ehci_pdata = {
+ .phy_init = s5p_usb_phy_init,
+ .phy_exit = s5p_usb_phy_exit,
+};
+
+static struct dwc3_exynos_data smdk5250_xhci_pdata = {
+ .phy_type = S5P_USB_PHY_DRD,
+ .phy_init = s5p_usb_phy_init,
+ .phy_exit = s5p_usb_phy_exit,
+};
+
+struct exynos_gpio_cfg {
+ unsigned int addr;
+ unsigned int num;
+ unsigned int bit;
+};
+
+static const char *rclksrc[] = {
+ [0] = "busclk",
+ [1] = "i2sclk",
+};
+
+static struct video_info smdk5250_dp_config = {
+ .name = "eDP-LVDS NXP PTN3460",
+
+ .h_sync_polarity = 0,
+ .v_sync_polarity = 0,
+ .interlaced = 0,
+
+ .color_space = COLOR_RGB,
+ .dynamic_range = VESA,
+ .ycbcr_coeff = COLOR_YCBCR601,
+ .color_depth = COLOR_8,
+
+ .link_rate = LINK_RATE_2_70GBPS,
+ .lane_count = LANE_COUNT2,
+};
+
+static struct exynos_dp_platdata smdk5250_dp_data __initdata = {
+ .video_info = &smdk5250_dp_config,
+ .training_type = HW_LINK_TRAINING,
+ .phy_init = s5p_dp_phy_init,
+ .phy_exit = s5p_dp_phy_exit,
+};
+
+static int exynos_cfg_i2s_gpio(struct platform_device *pdev)
+{
+ int id;
+ /* configure GPIO for i2s port */
+ struct exynos_gpio_cfg exynos5_cfg[3] = {
+ { EXYNOS5_GPZ(0), 7, S3C_GPIO_SFN(2) },
+ { EXYNOS5_GPB0(0), 5, S3C_GPIO_SFN(2) },
+ { EXYNOS5_GPB1(0), 5, S3C_GPIO_SFN(2) }
+ };
+
+ if (pdev->dev.of_node) {
+ id = of_alias_get_id(pdev->dev.of_node, "i2s");
+ if (id < 0)
+ dev_err(&pdev->dev, "failed to get alias id:%d\n", id);
+ } else {
+ id = pdev->id;
+ }
+
+ if (id < 0 || id > 2) {
+ printk(KERN_ERR "Invalid Device %d\n", id);
+ return -EINVAL;
+ }
+
+ s3c_gpio_cfgpin_range(exynos5_cfg[id].addr,
+ exynos5_cfg[id].num, exynos5_cfg[id].bit);
+
+ return 0;
+}
+
+static struct s3c_audio_pdata i2sv5_pdata = {
+ .cfg_gpio = exynos_cfg_i2s_gpio,
+ .type = {
+ .i2s = {
+ .quirks = QUIRK_PRI_6CHAN | QUIRK_SEC_DAI
+ | QUIRK_NEED_RSTCLR,
+ .src_clk = rclksrc,
+ .idma_addr = EXYNOS4_AUDSS_INT_MEM,
+ },
+ },
+};
/*
* The following lookup table is used to override device names when devices
* are registered from device tree. This is temporarily added to enable
"s3c2440-i2c.0", NULL),
OF_DEV_AUXDATA("samsung,s3c2440-i2c", EXYNOS5_PA_IIC(1),
"s3c2440-i2c.1", NULL),
+ OF_DEV_AUXDATA("samsung,s3c2440-i2c", EXYNOS5_PA_IIC(2),
+ "s3c2440-i2c.2", NULL),
+ OF_DEV_AUXDATA("samsung,s3c2440-i2c", EXYNOS5_PA_IIC(3),
+ "s3c2440-i2c.3", NULL),
+ OF_DEV_AUXDATA("samsung,s3c2440-i2c", EXYNOS5_PA_IIC(4),
+ "s3c2440-i2c.4", NULL),
+ OF_DEV_AUXDATA("samsung,s3c2440-i2c", EXYNOS5_PA_IIC(5),
+ "s3c2440-i2c.5", NULL),
+ OF_DEV_AUXDATA("samsung,s3c2440-i2c", EXYNOS5_PA_IIC(6),
+ "s3c2440-i2c.6", NULL),
+ OF_DEV_AUXDATA("samsung,s3c2440-i2c", EXYNOS5_PA_IIC(7),
+ "s3c2440-i2c.7", NULL),
+ OF_DEV_AUXDATA("samsung,s3c2440-i2c", EXYNOS5_PA_IIC(8),
+ "s3c2440-hdmiphy-i2c", NULL),
+ OF_DEV_AUXDATA("samsung,exynos4210-spi", EXYNOS5_PA_SPI0,
+ "exynos4210-spi.0", NULL),
+ OF_DEV_AUXDATA("samsung,exynos4210-spi", EXYNOS5_PA_SPI1,
+ "exynos4210-spi.1", NULL),
+ OF_DEV_AUXDATA("samsung,exynos4210-spi", EXYNOS5_PA_SPI2,
+ "exynos4210-spi.2", NULL),
OF_DEV_AUXDATA("synopsis,dw-mshc-exynos5250", 0x12200000,
"dw_mmc.0", NULL),
OF_DEV_AUXDATA("synopsis,dw-mshc-exynos5250", 0x12210000,
OF_DEV_AUXDATA("arm,pl330", EXYNOS5_PA_PDMA0, "dma-pl330.0", NULL),
OF_DEV_AUXDATA("arm,pl330", EXYNOS5_PA_PDMA1, "dma-pl330.1", NULL),
OF_DEV_AUXDATA("arm,pl330", EXYNOS5_PA_MDMA1, "dma-pl330.2", NULL),
+ OF_DEV_AUXDATA("samsung,s5p-sysmmu", 0x10A60000,
+ "s5p-sysmmu.2", &platdata_sysmmu_g2d),
+ OF_DEV_AUXDATA("samsung,s5p-sysmmu", 0x11210000,
+ "s5p-sysmmu.3", &platdata_sysmmu_mfc_l),
+ OF_DEV_AUXDATA("samsung,s5p-sysmmu", 0x11200000,
+ "s5p-sysmmu.4", &platdata_sysmmu_mfc_r),
+ OF_DEV_AUXDATA("samsung,s5p-sysmmu", 0x14640000,
+ "s5p-sysmmu.27", &platdata_sysmmu_fimd),
+ OF_DEV_AUXDATA("samsung,s5p-sysmmu", 0x14650000,
+ "s5p-sysmmu.28", &platdata_sysmmu_tv),
+ OF_DEV_AUXDATA("samsung,s5p-sysmmu", 0x13E80000,
+ "s5p-sysmmu.23", &platdata_sysmmu_gsc),
+ OF_DEV_AUXDATA("samsung,s5p-sysmmu", 0x13E90000,
+ "s5p-sysmmu.24", &platdata_sysmmu_gsc),
+ OF_DEV_AUXDATA("samsung,s5p-sysmmu", 0x13EA0000,
+ "s5p-sysmmu.25", &platdata_sysmmu_gsc),
+ OF_DEV_AUXDATA("samsung,s5p-sysmmu", 0x13EB0000,
+ "s5p-sysmmu.26", &platdata_sysmmu_gsc),
+ OF_DEV_AUXDATA("samsung,exynos5-fb", 0x14400000,
+ "exynos5-fb", &smdk5250_lcd1_pdata),
+ OF_DEV_AUXDATA("samsung,exynos5-mipi", 0x14500000,
+ "s5p-mipi-dsim", &dsim_platform_data),
+ OF_DEV_AUXDATA("samsung,exynos5-dp", 0x145B0000,
+ "s5p-dp", &smdk5250_dp_data),
+ OF_DEV_AUXDATA("samsung,s5p-mfc-v6", 0x11000000, "s5p-mfc-v6", NULL),
+ OF_DEV_AUXDATA("samsung,exynos-gsc", 0x13E00000,
+ "exynos-gsc.0", NULL),
+ OF_DEV_AUXDATA("samsung,exynos-gsc", 0x13E10000,
+ "exynos-gsc.1", NULL),
+ OF_DEV_AUXDATA("samsung,exynos-gsc", 0x13E20000,
+ "exynos-gsc.2", NULL),
+ OF_DEV_AUXDATA("samsung,exynos-gsc", 0x13E30000,
+ "exynos-gsc.3", NULL),
+#ifdef CONFIG_VIDEO_FIMG2D4X
+ OF_DEV_AUXDATA("samsung,s5p-g2d", 0x10850000,
+ "s5p-g2d", &fimg2d_data),
+#endif
+ OF_DEV_AUXDATA("samsung,exynos-ohci", 0x12120000,
+ "exynos-ohci", &smdk5250_ohci_pdata),
+ OF_DEV_AUXDATA("samsung,exynos-ehci", 0x12110000,
+ "s5p-ehci", &smdk5250_ehci_pdata),
+ OF_DEV_AUXDATA("samsung,exynos-xhci", 0x12000000,
+ "exynos-dwc3", &smdk5250_xhci_pdata),
+ OF_DEV_AUXDATA("samsung,i2s", 0x03830000,
+ "samsung-i2s.0", &i2sv5_pdata),
+ OF_DEV_AUXDATA("samsung,exynos5-hdmi", 0x14530000,
+ "exynos5-hdmi", NULL),
+ OF_DEV_AUXDATA("samsung,s5p-mixer", 0x14450000, "s5p-mixer", NULL),
{},
};
+static struct platform_device *smdk5250_devices[] __initdata = {
+ &smdk5250_lcd, /* for platform_lcd device */
+ &exynos_device_md0, /* for media device framework */
+ &exynos_device_md1, /* for media device framework */
+ &exynos_device_md2, /* for media device framework */
+ &samsung_asoc_dma, /* for audio dma interface device */
+ &exynos_drm_device,
+};
+
+static struct regulator_consumer_supply dummy_supplies[] = {
+ REGULATOR_SUPPLY("vddvario", "7000000.lan9215"),
+ REGULATOR_SUPPLY("vdd33a", "7000000.lan9215"),
+};
+
static void __init exynos5250_dt_map_io(void)
{
exynos_init_io(NULL, 0);
s3c24xx_init_clocks(24000000);
}
+static void __init exynos5_reserve(void)
+{
+ /* required to have enough address range to remap the IOMMU
+ * allocated buffers */
+ init_consistent_dma_size(SZ_64M);
+}
+
+static void s5p_tv_setup(void)
+{
+ /* direct HPD to HDMI chip */
+ gpio_request(EXYNOS5_GPX3(7), "hpd-plug");
+
+ gpio_direction_input(EXYNOS5_GPX3(7));
+ s3c_gpio_cfgpin(EXYNOS5_GPX3(7), S3C_GPIO_SFN(0x3));
+ s3c_gpio_setpull(EXYNOS5_GPX3(7), S3C_GPIO_PULL_NONE);
+}
+
+static void exynos5_i2c_setup(void)
+{
+ /* Setup the low-speed i2c controller interrupts */
+ writel(0x0, EXYNOS5_SYS_I2C_CFG);
+}
+
static void __init exynos5250_dt_machine_init(void)
{
- if (of_machine_is_compatible("samsung,smdk5250"))
- smsc911x_init(1);
+ struct device_node *srom_np, *np;
+
+ regulator_register_fixed(0, dummy_supplies, ARRAY_SIZE(dummy_supplies));
+
+ /* Setup pins for any SMSC 911x controller on the SROMC bus */
+ srom_np = of_find_node_by_path("/sromc-bus");
+ if (!srom_np) {
+ printk(KERN_ERR "No /sromc-bus property.\n");
+ goto out;
+ }
+ for_each_child_of_node(srom_np, np) {
+ if (of_device_is_compatible(np, "smsc,lan9115")) {
+ u32 reg;
+ of_property_read_u32(np, "reg", ®);
+ smsc911x_init(reg);
+ }
+ }
+
+ samsung_bl_set(&smdk5250_bl_gpio_info, &smdk5250_bl_data);
+
+ /*
+ * HACK ALERT! TODO: FIXME!
+ *
+ * We're going to hack in Daisy LCD info here for bringup purposes.
+ * Lots of things wrong with what we're doing here, but it works for
+ * bringup purposes.
+ */
+
+ if (of_machine_is_compatible("google,daisy")) {
+#ifdef CONFIG_DRM_EXYNOS_FIMD
+ smdk5250_lcd1_pdata.panel.timing.xres = 1366;
+ smdk5250_lcd1_pdata.panel.timing.yres = 768;
+ smdk5250_lcd1_pdata.panel_type = MIPI_LCD;
+#else
+ smdk5250_fb_win0.win_mode.xres = 1366;
+ smdk5250_fb_win0.win_mode.yres = 768;
+ smdk5250_fb_win0.virtual_x = 1366;
+ smdk5250_fb_win0.virtual_y = 768 * 2;
+
+ smdk5250_fb_win1.win_mode.xres = 1366;
+ smdk5250_fb_win1.win_mode.yres = 768;
+ smdk5250_fb_win1.virtual_x = 1366;
+ smdk5250_fb_win1.virtual_y = 768 * 2;
+
+ smdk5250_fb_win2.win_mode.xres = 1366;
+ smdk5250_fb_win2.win_mode.yres = 768;
+ smdk5250_fb_win2.virtual_x = 1366;
+ smdk5250_fb_win2.virtual_y = 768 * 2;
+#endif
+ dsim_lcd_info.lcd_size.width = 1366;
+ dsim_lcd_info.lcd_size.height = 768;
+ } else if (of_machine_is_compatible("google,snow")) {
+#ifdef CONFIG_DRM_EXYNOS_FIMD
+ smdk5250_lcd1_pdata.panel.timing = snow_fb_window;
+ smdk5250_lcd1_pdata.panel_type = DP_LCD;
+ smdk5250_lcd1_pdata.clock_rate = 267 * 1000 * 1000;
+ smdk5250_lcd1_pdata.vidcon1 = 0;
+#endif
+ }
+
+ if (gpio_request_one(EXYNOS5_GPX2(6), GPIOF_OUT_INIT_HIGH,
+ "HOST_VBUS_CONTROL")) {
+ printk(KERN_ERR "failed to request gpio_host_vbus\n");
+ } else {
+ s3c_gpio_setpull(EXYNOS5_GPX2(6), S3C_GPIO_PULL_NONE);
+ gpio_free(EXYNOS5_GPX2(6));
+ }
+
+ exynos5_i2c_setup();
+
+ /*
+ * BIG HACK: The wm8994 is not device tree enabled apparently, so
+ * needs to be added manually. ...but it's only on SMDK5250.
+ */
+ if (of_machine_is_compatible("samsung,smdk5250")) {
+ i2c_register_board_info(1, i2c_devs1, ARRAY_SIZE(i2c_devs1));
+ }
+
+ spi_register_board_info(spi1_board_info, ARRAY_SIZE(spi1_board_info));
of_platform_populate(NULL, of_default_bus_match_table,
exynos5250_auxdata_lookup, NULL);
+
+#ifdef CONFIG_DRM_EXYNOS_FIMD
+ if (of_machine_is_compatible("google,snow"))
+ exynos_dp_gpio_setup_24bpp();
+ else
+ exynos_fimd_gpio_setup_24bpp();
+#endif
+ s5p_tv_setup();
+
+ platform_add_devices(smdk5250_devices, ARRAY_SIZE(smdk5250_devices));
+out:
+ of_node_put(srom_np);
+ return;
}
static char const *exynos5250_dt_compat[] __initdata = {
DT_MACHINE_START(EXYNOS5_DT, "SAMSUNG EXYNOS5 (Flattened Device Tree)")
/* Maintainer: Kukjin Kim <kgene.kim@samsung.com> */
.init_irq = exynos5_init_irq,
+ .reserve = exynos5_reserve,
.map_io = exynos5250_dt_map_io,
.handle_irq = gic_handle_irq,
.init_machine = exynos5250_dt_machine_init,
--- /dev/null
+/* linux/arch/arm/mach-exynos/setup-dp.c
+ *
+ * Copyright (c) 2011 Samsung Electronics Co., Ltd.
+ * http://www.samsung.com/
+ *
+ * Base Samsung Exynos DP configuration
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#include <linux/io.h>
+#include <mach/regs-clock.h>
+
+void s5p_dp_phy_init(void)
+{
+ u32 reg;
+
+ /* Reference clock selection for DPTX_PHY: PAD_OSC_IN */
+ reg = __raw_readl(S3C_VA_SYS + 0x04d4);
+ reg &= ~(1 << 0);
+ __raw_writel(reg, S3C_VA_SYS + 0x04d4);
+
+ /* Select clock source for DPTX_PHY as XXTI */
+ reg = __raw_readl(S3C_VA_SYS + 0x04d8);
+ reg &= ~(1 << 3);
+ __raw_writel(reg, S3C_VA_SYS + 0x04d8);
+
+ reg = __raw_readl(S5P_DPTX_PHY_CONTROL);
+ reg |= S5P_DPTX_PHY_ENABLE;
+ __raw_writel(reg, S5P_DPTX_PHY_CONTROL);
+}
+
+void s5p_dp_phy_exit(void)
+{
+ u32 reg;
+
+ reg = __raw_readl(S5P_DPTX_PHY_CONTROL);
+ reg &= ~S5P_DPTX_PHY_ENABLE;
+ __raw_writel(reg, S5P_DPTX_PHY_CONTROL);
+}
--- /dev/null
+/* linux/arch/arm/mach-exynos/setup-fimd.c
+ *
+ * Copyright (c) 2009-2011 Samsung Electronics Co., Ltd.
+ * http://www.samsung.com
+ *
+ * Base Exynos4 FIMD configuration
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/fb.h>
+#include <linux/gpio.h>
+#include <linux/clk.h>
+
+#include <plat/fb.h>
+#include <plat/gpio-cfg.h>
+#include <plat/clock.h>
+
+#include <mach/regs-clock.h>
+#include <mach/map.h>
+
+void exynos4_fimd_cfg_gpios(unsigned int base, unsigned int nr,
+ unsigned int cfg, s5p_gpio_drvstr_t drvstr)
+{
+ s3c_gpio_cfgrange_nopull(base, nr, cfg);
+
+ for (; nr > 0; nr--, base++)
+ s5p_gpio_set_drvstr(base, drvstr);
+}
+
+int __init exynos4_fimd_setup_clock(struct device *dev, const char *bus_clk,
+ const char *parent, unsigned long clk_rate)
+{
+ struct clk *clk_parent;
+ struct clk *sclk;
+
+ sclk = clk_get(dev, bus_clk);
+ if (IS_ERR(sclk))
+ return PTR_ERR(sclk);
+
+ clk_parent = clk_get(NULL, parent);
+ if (IS_ERR(clk_parent)) {
+ clk_put(sclk);
+ return PTR_ERR(clk_parent);
+ }
+
+ if (clk_set_parent(sclk, clk_parent)) {
+ pr_err("Unable to set parent %s of clock %s.\n",
+ clk_parent->name, sclk->name);
+ clk_put(sclk);
+ clk_put(clk_parent);
+ return PTR_ERR(sclk);
+ }
+
+ if (!clk_rate)
+ clk_rate = 134000000UL;
+
+ if (clk_set_rate(sclk, clk_rate)) {
+ pr_err("%s rate change failed: %lu\n", sclk->name, clk_rate);
+ clk_put(sclk);
+ clk_put(clk_parent);
+ return PTR_ERR(sclk);
+ }
+
+ clk_put(sclk);
+ clk_put(clk_parent);
+
+ return 0;
+}
--- /dev/null
+/* linux/arch/arm/mach-exynos/setup-mipidsim.c
+ *
+ * Copyright (c) 2011 Samsung Electronics
+ *
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of
+ * the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * ERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+ * A 02111-1307 USA
+ */
+
+#include <linux/kernel.h>
+#include <linux/string.h>
+#include <linux/io.h>
+#include <linux/err.h>
+#include <linux/platform_device.h>
+#include <linux/clk.h>
+#include <linux/regulator/consumer.h>
+
+#include <mach/map.h>
+#include <mach/regs-clock.h>
+
+#include <plat/dsim.h>
+#include <plat/clock.h>
+#include <plat/regs-mipidsim.h>
+
+#define S5P_MIPI_M_RESETN 4
+
+static int s5p_dsim_enable_d_phy(struct mipi_dsim_device *dsim,
+ unsigned int enable)
+{
+ unsigned int reg;
+#if defined(CONFIG_ARCH_EXYNOS5)
+ reg = readl(S5P_MIPI_DPHY_CONTROL(1)) & ~(1 << 0);
+ reg |= (enable << 0);
+ writel(reg, S5P_MIPI_DPHY_CONTROL(1));
+#else
+ reg = readl(S5P_MIPI_DPHY_CONTROL(0)) & ~(1 << 0);
+ reg |= (enable << 0);
+ writel(reg, S5P_MIPI_DPHY_CONTROL(0));
+#endif
+ return 0;
+}
+
+static int s5p_dsim_enable_dsi_master(struct mipi_dsim_device *dsim,
+ unsigned int enable)
+{
+ unsigned int reg;
+#if defined(CONFIG_ARCH_EXYNOS5)
+ reg = readl(S5P_MIPI_DPHY_CONTROL(1)) & ~(1 << 2);
+ reg |= (enable << 2);
+ writel(reg, S5P_MIPI_DPHY_CONTROL(1));
+#else
+ reg = readl(S5P_MIPI_DPHY_CONTROL(0)) & ~(1 << 2);
+ reg |= (enable << 2);
+ writel(reg, S5P_MIPI_DPHY_CONTROL(0));
+#endif
+ return 0;
+}
+
+int s5p_dsim_part_reset(struct mipi_dsim_device *dsim)
+{
+#if defined(CONFIG_ARCH_EXYNOS5)
+ if (dsim->id == 0)
+ writel(S5P_MIPI_M_RESETN, S5P_MIPI_DPHY_CONTROL(1));
+#else
+ if (dsim->id == 0)
+ writel(S5P_MIPI_M_RESETN, S5P_MIPI_DPHY_CONTROL(0));
+#endif
+ return 0;
+}
+
+int s5p_dsim_init_d_phy(struct mipi_dsim_device *dsim, unsigned int enable)
+{
+ /**
+ * DPHY and aster block must be enabled at the system initialization
+ * step before data access from/to DPHY begins.
+ */
+ s5p_dsim_enable_d_phy(dsim, enable);
+
+ s5p_dsim_enable_dsi_master(dsim, enable);
+ return 0;
+}
*/
#include <linux/gpio.h>
-#include <linux/platform_device.h>
-
#include <plat/gpio-cfg.h>
-#include <plat/s3c64xx-spi.h>
#ifdef CONFIG_S3C64XX_DEV_SPI0
-struct s3c64xx_spi_info s3c64xx_spi0_pdata __initdata = {
- .fifo_lvl_mask = 0x1ff,
- .rx_lvl_offset = 15,
- .high_speed = 1,
- .clk_from_cmu = true,
- .tx_st_done = 25,
-};
-
-int s3c64xx_spi0_cfg_gpio(struct platform_device *dev)
+int s3c64xx_spi0_cfg_gpio(void)
{
s3c_gpio_cfgpin(EXYNOS4_GPB(0), S3C_GPIO_SFN(2));
s3c_gpio_setpull(EXYNOS4_GPB(0), S3C_GPIO_PULL_UP);
#endif
#ifdef CONFIG_S3C64XX_DEV_SPI1
-struct s3c64xx_spi_info s3c64xx_spi1_pdata __initdata = {
- .fifo_lvl_mask = 0x7f,
- .rx_lvl_offset = 15,
- .high_speed = 1,
- .clk_from_cmu = true,
- .tx_st_done = 25,
-};
-
-int s3c64xx_spi1_cfg_gpio(struct platform_device *dev)
+int s3c64xx_spi1_cfg_gpio(void)
{
s3c_gpio_cfgpin(EXYNOS4_GPB(4), S3C_GPIO_SFN(2));
s3c_gpio_setpull(EXYNOS4_GPB(4), S3C_GPIO_PULL_UP);
#endif
#ifdef CONFIG_S3C64XX_DEV_SPI2
-struct s3c64xx_spi_info s3c64xx_spi2_pdata __initdata = {
- .fifo_lvl_mask = 0x7f,
- .rx_lvl_offset = 15,
- .high_speed = 1,
- .clk_from_cmu = true,
- .tx_st_done = 25,
-};
-
-int s3c64xx_spi2_cfg_gpio(struct platform_device *dev)
+int s3c64xx_spi2_cfg_gpio(void)
{
s3c_gpio_cfgpin(EXYNOS4_GPC1(1), S3C_GPIO_SFN(5));
s3c_gpio_setpull(EXYNOS4_GPC1(1), S3C_GPIO_PULL_UP);
--- /dev/null
+/* linux/arch/arm/mach-exynos/setup-tvout.c
+ *
+ * Copyright (c) 2010 Samsung Electronics Co., Ltd.
+ * http://www.samsung.com/
+ *
+ * Base TVOUT gpio configuration
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/err.h>
+#include <linux/gpio.h>
+#include <linux/kernel.h>
+#include <linux/platform_device.h>
+#include <linux/types.h>
+#include <plat/clock.h>
+#include <plat/gpio-cfg.h>
+#include <mach/regs-clock.h>
+#include <mach/regs-gpio.h>
+#include <linux/io.h>
+#include <mach/map.h>
+#include <mach/gpio.h>
+#include <plat/tvout.h>
+#include <plat/cpu.h>
+
+#if defined(CONFIG_ARCH_EXYNOS4)
+#define HDMI_GPX(_nr) EXYNOS4_GPX3(_nr)
+#elif defined(CONFIG_ARCH_EXYNOS5)
+#define HDMI_GPX(_nr) EXYNOS5_GPX3(_nr)
+#endif
+
+struct platform_device; /* don't need the contents */
+
+void s5p_int_src_hdmi_hpd(struct platform_device *pdev)
+{
+ s3c_gpio_cfgpin(HDMI_GPX(7), S3C_GPIO_SFN(0x3));
+ s3c_gpio_setpull(HDMI_GPX(7), S3C_GPIO_PULL_DOWN);
+}
+
+void s5p_int_src_ext_hpd(struct platform_device *pdev)
+{
+ s3c_gpio_cfgpin(HDMI_GPX(7), S3C_GPIO_SFN(0xf));
+ s3c_gpio_setpull(HDMI_GPX(7), S3C_GPIO_PULL_DOWN);
+}
+
+int s5p_hpd_read_gpio(struct platform_device *pdev)
+{
+ return gpio_get_value(HDMI_GPX(7));
+}
+
+int s5p_v4l2_hpd_read_gpio(void)
+{
+ return gpio_get_value(HDMI_GPX(7));
+}
+
+void s5p_v4l2_int_src_hdmi_hpd(void)
+{
+ s3c_gpio_cfgpin(HDMI_GPX(7), S3C_GPIO_SFN(0x3));
+ s3c_gpio_setpull(HDMI_GPX(7), S3C_GPIO_PULL_DOWN);
+}
+
+void s5p_v4l2_int_src_ext_hpd(void)
+{
+ s3c_gpio_cfgpin(HDMI_GPX(7), S3C_GPIO_SFN(0xf));
+ s3c_gpio_setpull(HDMI_GPX(7), S3C_GPIO_PULL_DOWN);
+}
+
+void s5p_cec_cfg_gpio(struct platform_device *pdev)
+{
+ s3c_gpio_cfgpin(HDMI_GPX(6), S3C_GPIO_SFN(0x3));
+ s3c_gpio_setpull(HDMI_GPX(6), S3C_GPIO_PULL_NONE);
+}
+
+#ifdef CONFIG_VIDEO_EXYNOS_TV
+void s5p_tv_setup(void)
+{
+ int ret;
+
+ /* direct HPD to HDMI chip */
+ ret = gpio_request(HDMI_GPX(7), "hpd-plug");
+ if (ret)
+ printk(KERN_ERR "failed to request HPD-plug\n");
+ gpio_direction_input(HDMI_GPX(7));
+ s3c_gpio_cfgpin(HDMI_GPX(7), S3C_GPIO_SFN(0xf));
+ s3c_gpio_setpull(HDMI_GPX(7), S3C_GPIO_PULL_NONE);
+
+ /* HDMI CEC */
+ ret = gpio_request(HDMI_GPX(6), "hdmi-cec");
+ if (ret)
+ printk(KERN_ERR "failed to request HDMI-CEC\n");
+ gpio_direction_input(HDMI_GPX(6));
+ s3c_gpio_cfgpin(HDMI_GPX(6), S3C_GPIO_SFN(0x3));
+ s3c_gpio_setpull(HDMI_GPX(6), S3C_GPIO_PULL_NONE);
+}
+#endif
#include <mach/regs-usb-phy.h>
#include <plat/cpu.h>
#include <plat/usb-phy.h>
+#include <plat/regs-usb3-exynos-drd-phy.h>
+
+#define PHY_ENABLE 1
+#define PHY_DISABLE 0
+#define EXYNOS4_USB_CFG (S3C_VA_SYS + 0x21C)
+
+enum usb_phy_type {
+ USB_PHY = (0x1 << 0),
+ USB_PHY0 = (0x1 << 0),
+ USB_PHY1 = (0x1 << 1),
+ USB_PHY_HSIC0 = (0x1 << 1),
+ USB_PHY_HSIC1 = (0x1 << 2),
+};
+
+struct exynos_usb_phy {
+ u8 phy0_usage;
+ u8 phy1_usage;
+ u8 phy2_usage;
+ unsigned long flags;
+};
+
static atomic_t host_usage;
+static struct exynos_usb_phy usb_phy_control;
+static DEFINE_SPINLOCK(phy_lock);
+static struct clk *phy_clk;
static int exynos4_usb_host_phy_is_on(void)
{
return (readl(EXYNOS4_PHYPWR) & PHY1_STD_ANALOG_POWERDOWN) ? 0 : 1;
}
+static int exynos5_usb_host_phy20_is_on(void)
+{
+ return (readl(EXYNOS5_PHY_HOST_CTRL0) & HOST_CTRL0_PHYSWRSTALL) ? 0 : 1;
+}
+
+static int exynos_usb_phy_clock_enable(struct platform_device *pdev)
+{
+ int err;
+
+ if (!phy_clk) {
+ if (soc_is_exynos5250())
+ phy_clk = clk_get(&pdev->dev, "usbhost");
+
+ if (IS_ERR(phy_clk)) {
+ dev_err(&pdev->dev, "Failed to get phy clock\n");
+ return PTR_ERR(phy_clk);
+ }
+ }
+
+ err = clk_enable(phy_clk);
+
+ return err;
+}
+
+static void exynos_usb_mux_change(struct platform_device *pdev, int val)
+{
+ u32 is_host;
+ if (soc_is_exynos5250()) {
+ is_host = readl(EXYNOS5_USB_CFG);
+ writel(val, EXYNOS5_USB_CFG);
+ }
+ if (is_host != val)
+ dev_dbg(&pdev->dev, "Change USB MUX from %s to %s",
+ is_host ? "Host" : "Device", val ? "Host" : "Device");
+}
+
+static void exynos_usb_phy_control(enum usb_phy_type phy_type , int on)
+{
+ spin_lock(&phy_lock);
+ if (soc_is_exynos5250()) {
+ if (phy_type & USB_PHY0) {
+ if (on == PHY_ENABLE
+ && (usb_phy_control.phy0_usage++) == 0)
+ writel(S5P_USBDRD_PHY_ENABLE,
+ S5P_USBDRD_PHY_CONTROL);
+ else if (on == PHY_DISABLE
+ && (--usb_phy_control.phy0_usage) == 0)
+ writel(~S5P_USBDRD_PHY_ENABLE,
+ S5P_USBDRD_PHY_CONTROL);
+
+ } else if (phy_type & USB_PHY1) {
+ if (on == PHY_ENABLE
+ && (usb_phy_control.phy1_usage++) == 0)
+ writel(S5P_USBHOST_PHY_ENABLE,
+ S5P_USBHOST_PHY_CONTROL);
+ else if (on == PHY_DISABLE
+ && (--usb_phy_control.phy1_usage) == 0)
+ writel(~S5P_USBHOST_PHY_ENABLE,
+ S5P_USBHOST_PHY_CONTROL);
+ }
+ }
+ spin_unlock(&phy_lock);
+}
+
+static u32 exynos_usb_phy_set_clock(struct platform_device *pdev)
+{
+ struct clk *ref_clk;
+ u32 refclk_freq = 0;
+
+ ref_clk = clk_get(&pdev->dev, "ext_xtal");
+
+ if (IS_ERR(ref_clk)) {
+ dev_err(&pdev->dev, "Failed to get reference clock\n");
+ return PTR_ERR(ref_clk);
+ }
+ if (soc_is_exynos5250()) {
+ switch (clk_get_rate(ref_clk)) {
+ case 96 * 100000:
+ refclk_freq = EXYNOS5_CLKSEL_9600K;
+ break;
+ case 10 * MHZ:
+ refclk_freq = EXYNOS5_CLKSEL_10M;
+ break;
+ case 12 * MHZ:
+ refclk_freq = EXYNOS5_CLKSEL_12M;
+ break;
+ case 192 * 100000:
+ refclk_freq = EXYNOS5_CLKSEL_19200K;
+ break;
+ case 20 * MHZ:
+ refclk_freq = EXYNOS5_CLKSEL_20M;
+ break;
+ case 50 * MHZ:
+ refclk_freq = EXYNOS5_CLKSEL_50M;
+ break;
+ case 24 * MHZ:
+ default:
+ /* default reference clock */
+ refclk_freq = EXYNOS5_CLKSEL_24M;
+ break;
+ }
+ }
+ clk_put(ref_clk);
+
+ return refclk_freq;
+}
+
+static int exynos5_usb_phy30_init(struct platform_device *pdev)
+{
+ int ret;
+ u32 reg;
+ bool use_ext_clk = true;
+
+ ret = exynos_usb_phy_clock_enable(pdev);
+ if (ret)
+ return ret;
+
+ exynos_usb_phy_control(USB_PHY0, PHY_ENABLE);
+
+ /* Reset USB 3.0 PHY */
+ writel(0x00000000, EXYNOS_USB3_PHYREG0);
+ writel(0x24d4e6e4, EXYNOS_USB3_PHYPARAM0);
+ writel(0x03fff820, EXYNOS_USB3_PHYPARAM1);
+ writel(0x00000000, EXYNOS_USB3_PHYRESUME);
+
+ writel(0x08000000, EXYNOS_USB3_LINKSYSTEM);
+ writel(0x00000004, EXYNOS_USB3_PHYBATCHG);
+ /* REVISIT :use externel clock 100MHz */
+ if (use_ext_clk)
+ writel(readl(EXYNOS_USB3_PHYPARAM0) | (0x1<<31),
+ EXYNOS_USB3_PHYPARAM0);
+ else
+ writel(readl(EXYNOS_USB3_PHYPARAM0) & ~(0x1<<31),
+ EXYNOS_USB3_PHYPARAM0);
+
+ /* UTMI Power Control */
+ writel(EXYNOS_USB3_PHYUTMI_OTGDISABLE, EXYNOS_USB3_PHYUTMI);
+
+ /* Set 100MHz external clock */
+ reg = EXYNOS_USB3_PHYCLKRST_PORTRESET |
+ /* HS PLL uses ref_pad_clk{p,m} or ref_alt_clk_{p,m}
+ * as reference */
+ EXYNOS_USB3_PHYCLKRST_REFCLKSEL(2) |
+ /* Digital power supply in normal operating mode */
+ EXYNOS_USB3_PHYCLKRST_RETENABLEN |
+ /* 0x27-100MHz, 0x2a-24MHz, 0x31-20MHz, 0x38-19.2MHz */
+ EXYNOS_USB3_PHYCLKRST_FSEL(0x27) |
+ /* 0x19-100MHz, 0x68-24MHz, 0x7d-20Mhz */
+ EXYNOS_USB3_PHYCLKRST_MPLL_MULTIPLIER(0x19) |
+ /* Enable ref clock for SS function */
+ EXYNOS_USB3_PHYCLKRST_REF_SSP_EN |
+ /* Enable spread spectrum */
+ EXYNOS_USB3_PHYCLKRST_SSC_EN |
+ EXYNOS_USB3_PHYCLKRST_COMMONONN;
+
+ writel(reg, EXYNOS_USB3_PHYCLKRST);
+
+ udelay(10);
+
+ reg &= ~(EXYNOS_USB3_PHYCLKRST_PORTRESET);
+ writel(reg, EXYNOS_USB3_PHYCLKRST);
+
+ return 0;
+}
+
+static int exynos5_usb_phy30_exit(struct platform_device *pdev)
+{
+ u32 reg;
+
+ reg = EXYNOS_USB3_PHYUTMI_OTGDISABLE |
+ EXYNOS_USB3_PHYUTMI_FORCESUSPEND |
+ EXYNOS_USB3_PHYUTMI_FORCESLEEP;
+ writel(reg, EXYNOS_USB3_PHYUTMI);
+
+ exynos_usb_phy_control(USB_PHY0, PHY_DISABLE);
+
+ return 0;
+}
+
+static int exynos5_usb_phy20_init(struct platform_device *pdev)
+{
+ int ret;
+ u32 refclk_freq;
+ u32 hostphy_ctrl0, otgphy_sys, hsic_ctrl, ehcictrl;
+
+ atomic_inc(&host_usage);
+ ret = exynos_usb_phy_clock_enable(pdev);
+ if (ret)
+ return ret;
+
+ if (exynos5_usb_host_phy20_is_on()) {
+ dev_err(&pdev->dev, "Already power on PHY\n");
+ return 0;
+ }
+
+ exynos_usb_mux_change(pdev, 1);
+
+ exynos_usb_phy_control(USB_PHY1, PHY_ENABLE);
+
+ /* Host and Device should be set at the same time */
+ hostphy_ctrl0 = readl(EXYNOS5_PHY_HOST_CTRL0);
+ hostphy_ctrl0 &= ~(HOST_CTRL0_FSEL_MASK);
+ otgphy_sys = readl(EXYNOS5_PHY_OTG_SYS);
+ otgphy_sys &= ~(OTG_SYS_CTRL0_FSEL_MASK);
+
+ /* 2.0 phy reference clock configuration */
+ refclk_freq = exynos_usb_phy_set_clock(pdev);
+ hostphy_ctrl0 |= (refclk_freq << HOST_CTRL0_CLKSEL_SHIFT);
+ otgphy_sys |= (refclk_freq << OTG_SYS_CLKSEL_SHIFT);
+
+ /* COMMON Block configuration during suspend */
+ hostphy_ctrl0 &= ~(HOST_CTRL0_COMMONON_N);
+ otgphy_sys |= (OTG_SYS_COMMON_ON);
+
+ /* otg phy reset */
+ otgphy_sys &= ~(OTG_SYS_FORCE_SUSPEND | OTG_SYS_SIDDQ_UOTG
+ | OTG_SYS_FORCE_SLEEP);
+ otgphy_sys &= ~(OTG_SYS_REF_CLK_SEL_MASK);
+ otgphy_sys |= (OTG_SYS_REF_CLK_SEL(0x2) | OTG_SYS_OTGDISABLE);
+ otgphy_sys |= (OTG_SYS_PHY0_SW_RST | OTG_SYS_LINK_SW_RST_UOTG
+ | OTG_SYS_PHYLINK_SW_RESET);
+ writel(otgphy_sys, EXYNOS5_PHY_OTG_SYS);
+ udelay(10);
+ otgphy_sys &= ~(OTG_SYS_PHY0_SW_RST | OTG_SYS_LINK_SW_RST_UOTG
+ | OTG_SYS_PHYLINK_SW_RESET);
+ writel(otgphy_sys, EXYNOS5_PHY_OTG_SYS);
+ /* host phy reset */
+ hostphy_ctrl0 &= ~(HOST_CTRL0_PHYSWRST | HOST_CTRL0_PHYSWRSTALL
+ | HOST_CTRL0_SIDDQ);
+ hostphy_ctrl0 &= ~(HOST_CTRL0_FORCESUSPEND | HOST_CTRL0_FORCESLEEP);
+ hostphy_ctrl0 |= (HOST_CTRL0_LINKSWRST | HOST_CTRL0_UTMISWRST);
+ writel(hostphy_ctrl0, EXYNOS5_PHY_HOST_CTRL0);
+ udelay(10);
+ hostphy_ctrl0 &= ~(HOST_CTRL0_LINKSWRST | HOST_CTRL0_UTMISWRST);
+ writel(hostphy_ctrl0, EXYNOS5_PHY_HOST_CTRL0);
+
+ /* HSIC phy reset */
+ hsic_ctrl = (HSIC_CTRL_REFCLKDIV(0x24) | HSIC_CTRL_REFCLKSEL(0x2)
+ | HSIC_CTRL_PHYSWRST);
+ writel(hsic_ctrl, EXYNOS5_PHY_HSIC_CTRL1);
+ writel(hsic_ctrl, EXYNOS5_PHY_HSIC_CTRL2);
+ udelay(10);
+ hsic_ctrl &= ~(HSIC_CTRL_PHYSWRST);
+ writel(hsic_ctrl, EXYNOS5_PHY_HSIC_CTRL1);
+ writel(hsic_ctrl, EXYNOS5_PHY_HSIC_CTRL2);
+
+ udelay(80);
+
+ ehcictrl = readl(EXYNOS5_PHY_HOST_EHCICTRL);
+ ehcictrl |= (EHCICTRL_ENAINCRXALIGN | EHCICTRL_ENAINCR4
+ | EHCICTRL_ENAINCR8 | EHCICTRL_ENAINCR16);
+ writel(ehcictrl, EXYNOS5_PHY_HOST_EHCICTRL);
+
+ return 0;
+}
+static int exynos5_usb_phy20_exit(struct platform_device *pdev)
+{
+ u32 hostphy_ctrl0, otgphy_sys, hsic_ctrl;
+
+ if (atomic_dec_return(&host_usage) > 0) {
+ dev_info(&pdev->dev, "still being used\n");
+ return -EBUSY;
+ }
+
+ hsic_ctrl = (HSIC_CTRL_REFCLKDIV(0x24) | HSIC_CTRL_REFCLKSEL(0x2)
+ | HSIC_CTRL_SIDDQ | HSIC_CTRL_FORCESLEEP
+ | HSIC_CTRL_FORCESUSPEND);
+ writel(hsic_ctrl, EXYNOS5_PHY_HSIC_CTRL1);
+ writel(hsic_ctrl, EXYNOS5_PHY_HSIC_CTRL2);
+
+ hostphy_ctrl0 = readl(EXYNOS5_PHY_HOST_CTRL0);
+ hostphy_ctrl0 |= (HOST_CTRL0_SIDDQ);
+ hostphy_ctrl0 |= (HOST_CTRL0_FORCESUSPEND | HOST_CTRL0_FORCESLEEP);
+ hostphy_ctrl0 |= (HOST_CTRL0_PHYSWRST | HOST_CTRL0_PHYSWRSTALL);
+ writel(hostphy_ctrl0, EXYNOS5_PHY_HOST_CTRL0);
+
+ otgphy_sys = readl(EXYNOS5_PHY_OTG_SYS);
+ otgphy_sys |= (OTG_SYS_FORCE_SUSPEND | OTG_SYS_SIDDQ_UOTG
+ | OTG_SYS_FORCE_SLEEP);
+ writel(otgphy_sys, EXYNOS5_PHY_OTG_SYS);
+
+ exynos_usb_phy_control(USB_PHY1, PHY_DISABLE);
+
+ return 0;
+}
+
+
static int exynos4_usb_phy1_init(struct platform_device *pdev)
{
struct clk *otg_clk;
int s5p_usb_phy_init(struct platform_device *pdev, int type)
{
- if (type == S5P_USB_PHY_HOST)
- return exynos4_usb_phy1_init(pdev);
-
+ if (type == S5P_USB_PHY_HOST) {
+ if (soc_is_exynos5250())
+ return exynos5_usb_phy20_init(pdev);
+ else
+ return exynos4_usb_phy1_init(pdev);
+ } else if (type == S5P_USB_PHY_DRD) {
+ if (soc_is_exynos5250())
+ return exynos5_usb_phy30_init(pdev);
+ else
+ dev_err(&pdev->dev, "USB 3.0 DRD not present\n");
+ }
return -EINVAL;
}
int s5p_usb_phy_exit(struct platform_device *pdev, int type)
{
- if (type == S5P_USB_PHY_HOST)
- return exynos4_usb_phy1_exit(pdev);
-
+ if (type == S5P_USB_PHY_HOST) {
+ if (soc_is_exynos5250())
+ return exynos5_usb_phy20_exit(pdev);
+ else
+ return exynos4_usb_phy1_exit(pdev);
+ } else if (type == S5P_USB_PHY_DRD) {
+ if (soc_is_exynos5250())
+ return exynos5_usb_phy30_exit(pdev);
+ else
+ dev_err(&pdev->dev, "USB 3.0 DRD not present\n");
+ }
return -EINVAL;
}
.ctrlbit = S3C_CLKCON_PCLK_KEYPAD,
}, {
.name = "spi",
- .devname = "s3c64xx-spi.0",
+ .devname = "s3c6410-spi.0",
.parent = &clk_p,
.enable = s3c64xx_pclk_ctrl,
.ctrlbit = S3C_CLKCON_PCLK_SPI0,
}, {
.name = "spi",
- .devname = "s3c64xx-spi.1",
+ .devname = "s3c6410-spi.1",
.parent = &clk_p,
.enable = s3c64xx_pclk_ctrl,
.ctrlbit = S3C_CLKCON_PCLK_SPI1,
static struct clk clk_48m_spi0 = {
.name = "spi_48m",
- .devname = "s3c64xx-spi.0",
+ .devname = "s3c6410-spi.0",
.parent = &clk_48m,
.enable = s3c64xx_sclk_ctrl,
.ctrlbit = S3C_CLKCON_SCLK_SPI0_48,
static struct clk clk_48m_spi1 = {
.name = "spi_48m",
- .devname = "s3c64xx-spi.1",
+ .devname = "s3c6410-spi.1",
.parent = &clk_48m,
.enable = s3c64xx_sclk_ctrl,
.ctrlbit = S3C_CLKCON_SCLK_SPI1_48,
static struct clksrc_clk clk_sclk_spi0 = {
.clk = {
.name = "spi-bus",
- .devname = "s3c64xx-spi.0",
+ .devname = "s3c6410-spi.0",
.ctrlbit = S3C_CLKCON_SCLK_SPI0,
.enable = s3c64xx_sclk_ctrl,
},
static struct clksrc_clk clk_sclk_spi1 = {
.clk = {
.name = "spi-bus",
- .devname = "s3c64xx-spi.1",
+ .devname = "s3c6410-spi.1",
.ctrlbit = S3C_CLKCON_SCLK_SPI1,
.enable = s3c64xx_sclk_ctrl,
},
CLKDEV_INIT("s3c-sdhci.1", "mmc_busclk.2", &clk_sclk_mmc1.clk),
CLKDEV_INIT("s3c-sdhci.2", "mmc_busclk.2", &clk_sclk_mmc2.clk),
CLKDEV_INIT(NULL, "spi_busclk0", &clk_p),
- CLKDEV_INIT("s3c64xx-spi.0", "spi_busclk1", &clk_sclk_spi0.clk),
- CLKDEV_INIT("s3c64xx-spi.0", "spi_busclk2", &clk_48m_spi0),
- CLKDEV_INIT("s3c64xx-spi.1", "spi_busclk1", &clk_sclk_spi1.clk),
- CLKDEV_INIT("s3c64xx-spi.1", "spi_busclk2", &clk_48m_spi1),
+ CLKDEV_INIT("s3c6410-spi.0", "spi_busclk1", &clk_sclk_spi0.clk),
+ CLKDEV_INIT("s3c6410-spi.0", "spi_busclk2", &clk_48m_spi0),
+ CLKDEV_INIT("s3c6410-spi.1", "spi_busclk1", &clk_sclk_spi1.clk),
+ CLKDEV_INIT("s3c6410-spi.1", "spi_busclk2", &clk_48m_spi1),
};
#define GET_DIV(clk, field) ((((clk) & field##_MASK) >> field##_SHIFT) + 1)
i2c_register_board_info(1, i2c_devs1, ARRAY_SIZE(i2c_devs1));
samsung_keypad_set_platdata(&crag6410_keypad_data);
- s3c64xx_spi0_set_platdata(&s3c64xx_spi0_pdata, 0, 1);
+ s3c64xx_spi0_set_platdata(NULL, 0, 1);
platform_add_devices(crag6410_devices, ARRAY_SIZE(crag6410_devices));
*/
#include <linux/gpio.h>
-#include <linux/platform_device.h>
-
#include <plat/gpio-cfg.h>
-#include <plat/s3c64xx-spi.h>
#ifdef CONFIG_S3C64XX_DEV_SPI0
-struct s3c64xx_spi_info s3c64xx_spi0_pdata __initdata = {
- .fifo_lvl_mask = 0x7f,
- .rx_lvl_offset = 13,
- .tx_st_done = 21,
-};
-
-int s3c64xx_spi0_cfg_gpio(struct platform_device *dev)
+int s3c64xx_spi0_cfg_gpio(void)
{
s3c_gpio_cfgall_range(S3C64XX_GPC(0), 3,
S3C_GPIO_SFN(2), S3C_GPIO_PULL_UP);
#endif
#ifdef CONFIG_S3C64XX_DEV_SPI1
-struct s3c64xx_spi_info s3c64xx_spi1_pdata __initdata = {
- .fifo_lvl_mask = 0x7f,
- .rx_lvl_offset = 13,
- .tx_st_done = 21,
-};
-
-int s3c64xx_spi1_cfg_gpio(struct platform_device *dev)
+int s3c64xx_spi1_cfg_gpio(void)
{
s3c_gpio_cfgall_range(S3C64XX_GPC(4), 3,
S3C_GPIO_SFN(2), S3C_GPIO_PULL_UP);
.ctrlbit = (1 << 17),
}, {
.name = "spi",
- .devname = "s3c64xx-spi.0",
+ .devname = "s5p64x0-spi.0",
.parent = &clk_pclk_low.clk,
.enable = s5p64x0_pclk_ctrl,
.ctrlbit = (1 << 21),
}, {
.name = "spi",
- .devname = "s3c64xx-spi.1",
+ .devname = "s5p64x0-spi.1",
.parent = &clk_pclk_low.clk,
.enable = s5p64x0_pclk_ctrl,
.ctrlbit = (1 << 22),
static struct clksrc_clk clk_sclk_spi0 = {
.clk = {
.name = "sclk_spi",
- .devname = "s3c64xx-spi.0",
+ .devname = "s5p64x0-spi.0",
.ctrlbit = (1 << 20),
.enable = s5p64x0_sclk_ctrl,
},
static struct clksrc_clk clk_sclk_spi1 = {
.clk = {
.name = "sclk_spi",
- .devname = "s3c64xx-spi.1",
+ .devname = "s5p64x0-spi.1",
.ctrlbit = (1 << 21),
.enable = s5p64x0_sclk_ctrl,
},
CLKDEV_INIT(NULL, "clk_uart_baud2", &clk_pclk_low.clk),
CLKDEV_INIT(NULL, "clk_uart_baud3", &clk_sclk_uclk.clk),
CLKDEV_INIT(NULL, "spi_busclk0", &clk_p),
- CLKDEV_INIT("s3c64xx-spi.0", "spi_busclk1", &clk_sclk_spi0.clk),
- CLKDEV_INIT("s3c64xx-spi.1", "spi_busclk1", &clk_sclk_spi1.clk),
+ CLKDEV_INIT("s5p64x0-spi.0", "spi_busclk1", &clk_sclk_spi0.clk),
+ CLKDEV_INIT("s5p64x0-spi.1", "spi_busclk1", &clk_sclk_spi1.clk),
CLKDEV_INIT("s3c-sdhci.0", "mmc_busclk.2", &clk_sclk_mmc0.clk),
CLKDEV_INIT("s3c-sdhci.1", "mmc_busclk.2", &clk_sclk_mmc1.clk),
CLKDEV_INIT("s3c-sdhci.2", "mmc_busclk.2", &clk_sclk_mmc2.clk),
.ctrlbit = (1 << 17),
}, {
.name = "spi",
- .devname = "s3c64xx-spi.0",
+ .devname = "s5p64x0-spi.0",
.parent = &clk_pclk_low.clk,
.enable = s5p64x0_pclk_ctrl,
.ctrlbit = (1 << 21),
}, {
.name = "spi",
- .devname = "s3c64xx-spi.1",
+ .devname = "s5p64x0-spi.1",
.parent = &clk_pclk_low.clk,
.enable = s5p64x0_pclk_ctrl,
.ctrlbit = (1 << 22),
static struct clksrc_clk clk_sclk_spi0 = {
.clk = {
.name = "sclk_spi",
- .devname = "s3c64xx-spi.0",
+ .devname = "s5p64x0-spi.0",
.ctrlbit = (1 << 20),
.enable = s5p64x0_sclk_ctrl,
},
static struct clksrc_clk clk_sclk_spi1 = {
.clk = {
.name = "sclk_spi",
- .devname = "s3c64xx-spi.1",
+ .devname = "s5p64x0-spi.1",
.ctrlbit = (1 << 21),
.enable = s5p64x0_sclk_ctrl,
},
CLKDEV_INIT(NULL, "clk_uart_baud2", &clk_pclk_low.clk),
CLKDEV_INIT(NULL, "clk_uart_baud3", &clk_sclk_uclk.clk),
CLKDEV_INIT(NULL, "spi_busclk0", &clk_p),
- CLKDEV_INIT("s3c64xx-spi.0", "spi_busclk1", &clk_sclk_spi0.clk),
- CLKDEV_INIT("s3c64xx-spi.1", "spi_busclk1", &clk_sclk_spi1.clk),
+ CLKDEV_INIT("s5p64x0-spi.0", "spi_busclk1", &clk_sclk_spi0.clk),
+ CLKDEV_INIT("s5p64x0-spi.1", "spi_busclk1", &clk_sclk_spi1.clk),
CLKDEV_INIT("s3c-sdhci.0", "mmc_busclk.2", &clk_sclk_mmc0.clk),
CLKDEV_INIT("s3c-sdhci.1", "mmc_busclk.2", &clk_sclk_mmc1.clk),
CLKDEV_INIT("s3c-sdhci.2", "mmc_busclk.2", &clk_sclk_mmc2.clk),
*/
#include <linux/gpio.h>
-#include <linux/platform_device.h>
-#include <linux/io.h>
-
#include <plat/gpio-cfg.h>
-#include <plat/cpu.h>
-#include <plat/s3c64xx-spi.h>
#ifdef CONFIG_S3C64XX_DEV_SPI0
-struct s3c64xx_spi_info s3c64xx_spi0_pdata __initdata = {
- .fifo_lvl_mask = 0x1ff,
- .rx_lvl_offset = 15,
- .tx_st_done = 25,
-};
-
-int s3c64xx_spi0_cfg_gpio(struct platform_device *dev)
+int s3c64xx_spi0_cfg_gpio(void)
{
if (soc_is_s5p6450())
s3c_gpio_cfgall_range(S5P6450_GPC(0), 3,
#endif
#ifdef CONFIG_S3C64XX_DEV_SPI1
-struct s3c64xx_spi_info s3c64xx_spi1_pdata __initdata = {
- .fifo_lvl_mask = 0x7f,
- .rx_lvl_offset = 15,
- .tx_st_done = 25,
-};
-
-int s3c64xx_spi1_cfg_gpio(struct platform_device *dev)
+int s3c64xx_spi1_cfg_gpio(void)
{
if (soc_is_s5p6450())
s3c_gpio_cfgall_range(S5P6450_GPC(4), 3,
.ctrlbit = (1 << 5),
}, {
.name = "spi",
- .devname = "s3c64xx-spi.0",
+ .devname = "s5pc100-spi.0",
.parent = &clk_div_d1_bus.clk,
.enable = s5pc100_d1_4_ctrl,
.ctrlbit = (1 << 6),
}, {
.name = "spi",
- .devname = "s3c64xx-spi.1",
+ .devname = "s5pc100-spi.1",
.parent = &clk_div_d1_bus.clk,
.enable = s5pc100_d1_4_ctrl,
.ctrlbit = (1 << 7),
}, {
.name = "spi",
- .devname = "s3c64xx-spi.2",
+ .devname = "s5pc100-spi.2",
.parent = &clk_div_d1_bus.clk,
.enable = s5pc100_d1_4_ctrl,
.ctrlbit = (1 << 8),
static struct clk clk_48m_spi0 = {
.name = "spi_48m",
- .devname = "s3c64xx-spi.0",
+ .devname = "s5pc100-spi.0",
.parent = &clk_mout_48m.clk,
.enable = s5pc100_sclk0_ctrl,
.ctrlbit = (1 << 7),
static struct clk clk_48m_spi1 = {
.name = "spi_48m",
- .devname = "s3c64xx-spi.1",
+ .devname = "s5pc100-spi.1",
.parent = &clk_mout_48m.clk,
.enable = s5pc100_sclk0_ctrl,
.ctrlbit = (1 << 8),
static struct clk clk_48m_spi2 = {
.name = "spi_48m",
- .devname = "s3c64xx-spi.2",
+ .devname = "s5pc100-spi.2",
.parent = &clk_mout_48m.clk,
.enable = s5pc100_sclk0_ctrl,
.ctrlbit = (1 << 9),
static struct clksrc_clk clk_sclk_spi0 = {
.clk = {
.name = "sclk_spi",
- .devname = "s3c64xx-spi.0",
+ .devname = "s5pc100-spi.0",
.ctrlbit = (1 << 4),
.enable = s5pc100_sclk0_ctrl,
},
static struct clksrc_clk clk_sclk_spi1 = {
.clk = {
.name = "sclk_spi",
- .devname = "s3c64xx-spi.1",
+ .devname = "s5pc100-spi.1",
.ctrlbit = (1 << 5),
.enable = s5pc100_sclk0_ctrl,
},
static struct clksrc_clk clk_sclk_spi2 = {
.clk = {
.name = "sclk_spi",
- .devname = "s3c64xx-spi.2",
+ .devname = "s5pc100-spi.2",
.ctrlbit = (1 << 6),
.enable = s5pc100_sclk0_ctrl,
},
CLKDEV_INIT("s3c-sdhci.1", "mmc_busclk.2", &clk_sclk_mmc1.clk),
CLKDEV_INIT("s3c-sdhci.2", "mmc_busclk.2", &clk_sclk_mmc2.clk),
CLKDEV_INIT(NULL, "spi_busclk0", &clk_p),
- CLKDEV_INIT("s3c64xx-spi.0", "spi_busclk1", &clk_48m_spi0),
- CLKDEV_INIT("s3c64xx-spi.0", "spi_busclk2", &clk_sclk_spi0.clk),
- CLKDEV_INIT("s3c64xx-spi.1", "spi_busclk1", &clk_48m_spi1),
- CLKDEV_INIT("s3c64xx-spi.1", "spi_busclk2", &clk_sclk_spi1.clk),
- CLKDEV_INIT("s3c64xx-spi.2", "spi_busclk1", &clk_48m_spi2),
- CLKDEV_INIT("s3c64xx-spi.2", "spi_busclk2", &clk_sclk_spi2.clk),
+ CLKDEV_INIT("s5pc100-spi.0", "spi_busclk1", &clk_48m_spi0),
+ CLKDEV_INIT("s5pc100-spi.0", "spi_busclk2", &clk_sclk_spi0.clk),
+ CLKDEV_INIT("s5pc100-spi.1", "spi_busclk1", &clk_48m_spi1),
+ CLKDEV_INIT("s5pc100-spi.1", "spi_busclk2", &clk_sclk_spi1.clk),
+ CLKDEV_INIT("s5pc100-spi.2", "spi_busclk1", &clk_48m_spi2),
+ CLKDEV_INIT("s5pc100-spi.2", "spi_busclk2", &clk_sclk_spi2.clk),
};
void __init s5pc100_register_clocks(void)
*/
#include <linux/gpio.h>
-#include <linux/platform_device.h>
-
#include <plat/gpio-cfg.h>
-#include <plat/s3c64xx-spi.h>
#ifdef CONFIG_S3C64XX_DEV_SPI0
-struct s3c64xx_spi_info s3c64xx_spi0_pdata __initdata = {
- .fifo_lvl_mask = 0x7f,
- .rx_lvl_offset = 13,
- .high_speed = 1,
- .tx_st_done = 21,
-};
-
-int s3c64xx_spi0_cfg_gpio(struct platform_device *dev)
+int s3c64xx_spi0_cfg_gpio(void)
{
s3c_gpio_cfgall_range(S5PC100_GPB(0), 3,
S3C_GPIO_SFN(2), S3C_GPIO_PULL_UP);
#endif
#ifdef CONFIG_S3C64XX_DEV_SPI1
-struct s3c64xx_spi_info s3c64xx_spi1_pdata __initdata = {
- .fifo_lvl_mask = 0x7f,
- .rx_lvl_offset = 13,
- .high_speed = 1,
- .tx_st_done = 21,
-};
-
-int s3c64xx_spi1_cfg_gpio(struct platform_device *dev)
+int s3c64xx_spi1_cfg_gpio(void)
{
s3c_gpio_cfgall_range(S5PC100_GPB(4), 3,
S3C_GPIO_SFN(2), S3C_GPIO_PULL_UP);
#endif
#ifdef CONFIG_S3C64XX_DEV_SPI2
-struct s3c64xx_spi_info s3c64xx_spi2_pdata __initdata = {
- .fifo_lvl_mask = 0x7f,
- .rx_lvl_offset = 13,
- .high_speed = 1,
- .tx_st_done = 21,
-};
-
-int s3c64xx_spi2_cfg_gpio(struct platform_device *dev)
+int s3c64xx_spi2_cfg_gpio(void)
{
s3c_gpio_cfgpin(S5PC100_GPG3(0), S3C_GPIO_SFN(3));
s3c_gpio_setpull(S5PC100_GPG3(0), S3C_GPIO_PULL_UP);
.ctrlbit = (1 << 11),
}, {
.name = "spi",
- .devname = "s3c64xx-spi.0",
+ .devname = "s5pv210-spi.0",
.parent = &clk_pclk_psys.clk,
.enable = s5pv210_clk_ip3_ctrl,
.ctrlbit = (1<<12),
}, {
.name = "spi",
- .devname = "s3c64xx-spi.1",
+ .devname = "s5pv210-spi.1",
.parent = &clk_pclk_psys.clk,
.enable = s5pv210_clk_ip3_ctrl,
.ctrlbit = (1<<13),
}, {
.name = "spi",
- .devname = "s3c64xx-spi.2",
+ .devname = "s5pv210-spi.2",
.parent = &clk_pclk_psys.clk,
.enable = s5pv210_clk_ip3_ctrl,
.ctrlbit = (1<<14),
static struct clksrc_clk clk_sclk_spi0 = {
.clk = {
.name = "sclk_spi",
- .devname = "s3c64xx-spi.0",
+ .devname = "s5pv210-spi.0",
.enable = s5pv210_clk_mask0_ctrl,
.ctrlbit = (1 << 16),
},
static struct clksrc_clk clk_sclk_spi1 = {
.clk = {
.name = "sclk_spi",
- .devname = "s3c64xx-spi.1",
+ .devname = "s5pv210-spi.1",
.enable = s5pv210_clk_mask0_ctrl,
.ctrlbit = (1 << 17),
},
CLKDEV_INIT("s3c-sdhci.2", "mmc_busclk.2", &clk_sclk_mmc2.clk),
CLKDEV_INIT("s3c-sdhci.3", "mmc_busclk.2", &clk_sclk_mmc3.clk),
CLKDEV_INIT(NULL, "spi_busclk0", &clk_p),
- CLKDEV_INIT("s3c64xx-spi.0", "spi_busclk1", &clk_sclk_spi0.clk),
- CLKDEV_INIT("s3c64xx-spi.1", "spi_busclk1", &clk_sclk_spi1.clk),
+ CLKDEV_INIT("s5pv210-spi.0", "spi_busclk1", &clk_sclk_spi0.clk),
+ CLKDEV_INIT("s5pv210-spi.1", "spi_busclk1", &clk_sclk_spi1.clk),
};
void __init s5pv210_register_clocks(void)
*/
#include <linux/gpio.h>
-#include <linux/platform_device.h>
-
#include <plat/gpio-cfg.h>
-#include <plat/s3c64xx-spi.h>
#ifdef CONFIG_S3C64XX_DEV_SPI0
-struct s3c64xx_spi_info s3c64xx_spi0_pdata = {
- .fifo_lvl_mask = 0x1ff,
- .rx_lvl_offset = 15,
- .high_speed = 1,
- .tx_st_done = 25,
-};
-
-int s3c64xx_spi0_cfg_gpio(struct platform_device *dev)
+int s3c64xx_spi0_cfg_gpio(void)
{
s3c_gpio_cfgpin(S5PV210_GPB(0), S3C_GPIO_SFN(2));
s3c_gpio_setpull(S5PV210_GPB(0), S3C_GPIO_PULL_UP);
#endif
#ifdef CONFIG_S3C64XX_DEV_SPI1
-struct s3c64xx_spi_info s3c64xx_spi1_pdata = {
- .fifo_lvl_mask = 0x7f,
- .rx_lvl_offset = 15,
- .high_speed = 1,
- .tx_st_done = 25,
-};
-
-int s3c64xx_spi1_cfg_gpio(struct platform_device *dev)
+int s3c64xx_spi1_cfg_gpio(void)
{
s3c_gpio_cfgpin(S5PV210_GPB(4), S3C_GPIO_SFN(2));
s3c_gpio_setpull(S5PV210_GPB(4), S3C_GPIO_PULL_UP);
#include <linux/init.h>
#include <linux/device.h>
#include <linux/dma-mapping.h>
+#include <linux/dma-contiguous.h>
#include <linux/highmem.h>
+#include <linux/memblock.h>
#include <linux/slab.h>
+#include <linux/iommu.h>
+#include <linux/io.h>
+#include <linux/vmalloc.h>
#include <asm/memory.h>
#include <asm/highmem.h>
#include <asm/tlbflush.h>
#include <asm/sizes.h>
#include <asm/mach/arch.h>
+#include <asm/dma-iommu.h>
+#include <asm/mach/map.h>
+#include <asm/system_info.h>
+#include <asm/dma-contiguous.h>
#include "mm.h"
+/*
+ * The DMA API is built upon the notion of "buffer ownership". A buffer
+ * is either exclusively owned by the CPU (and therefore may be accessed
+ * by it) or exclusively owned by the DMA device. These helper functions
+ * represent the transitions between these two ownership states.
+ *
+ * Note, however, that on later ARMs, this notion does not work due to
+ * speculative prefetches. We model our approach on the assumption that
+ * the CPU does do speculative prefetches, which means we clean caches
+ * before transfers and delay cache invalidation until transfer completion.
+ *
+ */
+static void __dma_page_cpu_to_dev(struct page *, unsigned long,
+ size_t, enum dma_data_direction);
+static void __dma_page_dev_to_cpu(struct page *, unsigned long,
+ size_t, enum dma_data_direction);
+
+/**
+ * arm_dma_map_page - map a portion of a page for streaming DMA
+ * @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
+ * @page: page that buffer resides in
+ * @offset: offset into page for start of buffer
+ * @size: size of buffer to map
+ * @dir: DMA transfer direction
+ *
+ * Ensure that any data held in the cache is appropriately discarded
+ * or written back.
+ *
+ * The device owns this memory once this call has completed. The CPU
+ * can regain ownership by calling dma_unmap_page().
+ */
+static dma_addr_t arm_dma_map_page(struct device *dev, struct page *page,
+ unsigned long offset, size_t size, enum dma_data_direction dir,
+ struct dma_attrs *attrs)
+{
+ if (!arch_is_coherent() && !dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs))
+ __dma_page_cpu_to_dev(page, offset, size, dir);
+ return pfn_to_dma(dev, page_to_pfn(page)) + offset;
+}
+
+/**
+ * arm_dma_unmap_page - unmap a buffer previously mapped through dma_map_page()
+ * @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
+ * @handle: DMA address of buffer
+ * @size: size of buffer (same as passed to dma_map_page)
+ * @dir: DMA transfer direction (same as passed to dma_map_page)
+ *
+ * Unmap a page streaming mode DMA translation. The handle and size
+ * must match what was provided in the previous dma_map_page() call.
+ * All other usages are undefined.
+ *
+ * After this call, reads by the CPU to the buffer are guaranteed to see
+ * whatever the device wrote there.
+ */
+static void arm_dma_unmap_page(struct device *dev, dma_addr_t handle,
+ size_t size, enum dma_data_direction dir,
+ struct dma_attrs *attrs)
+{
+ if (!arch_is_coherent() && !dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs))
+ __dma_page_dev_to_cpu(pfn_to_page(dma_to_pfn(dev, handle)),
+ handle & ~PAGE_MASK, size, dir);
+}
+
+static void arm_dma_sync_single_for_cpu(struct device *dev,
+ dma_addr_t handle, size_t size, enum dma_data_direction dir)
+{
+ unsigned int offset = handle & (PAGE_SIZE - 1);
+ struct page *page = pfn_to_page(dma_to_pfn(dev, handle-offset));
+ if (!arch_is_coherent())
+ __dma_page_dev_to_cpu(page, offset, size, dir);
+}
+
+static void arm_dma_sync_single_for_device(struct device *dev,
+ dma_addr_t handle, size_t size, enum dma_data_direction dir)
+{
+ unsigned int offset = handle & (PAGE_SIZE - 1);
+ struct page *page = pfn_to_page(dma_to_pfn(dev, handle-offset));
+ if (!arch_is_coherent())
+ __dma_page_cpu_to_dev(page, offset, size, dir);
+}
+
+static int arm_dma_set_mask(struct device *dev, u64 dma_mask);
+
+struct dma_map_ops arm_dma_ops = {
+ .alloc = arm_dma_alloc,
+ .free = arm_dma_free,
+ .mmap = arm_dma_mmap,
+ .get_sgtable = arm_dma_get_sgtable,
+ .map_page = arm_dma_map_page,
+ .unmap_page = arm_dma_unmap_page,
+ .map_sg = arm_dma_map_sg,
+ .unmap_sg = arm_dma_unmap_sg,
+ .sync_single_for_cpu = arm_dma_sync_single_for_cpu,
+ .sync_single_for_device = arm_dma_sync_single_for_device,
+ .sync_sg_for_cpu = arm_dma_sync_sg_for_cpu,
+ .sync_sg_for_device = arm_dma_sync_sg_for_device,
+ .set_dma_mask = arm_dma_set_mask,
+};
+EXPORT_SYMBOL(arm_dma_ops);
+
static u64 get_coherent_dma_mask(struct device *dev)
{
u64 mask = (u64)arm_dma_limit;
return mask;
}
+static void __dma_clear_buffer(struct page *page, size_t size)
+{
+ void *ptr;
+ /*
+ * Ensure that the allocated pages are zeroed, and that any data
+ * lurking in the kernel direct-mapped region is invalidated.
+ */
+ ptr = page_address(page);
+ if (ptr) {
+ memset(ptr, 0, size);
+ dmac_flush_range(ptr, ptr + size);
+ outer_flush_range(__pa(ptr), __pa(ptr) + size);
+ }
+}
+
/*
* Allocate a DMA buffer for 'dev' of size 'size' using the
* specified gfp mask. Note that 'size' must be page aligned.
{
unsigned long order = get_order(size);
struct page *page, *p, *e;
- void *ptr;
- u64 mask = get_coherent_dma_mask(dev);
-
-#ifdef CONFIG_DMA_API_DEBUG
- u64 limit = (mask + 1) & ~mask;
- if (limit && size >= limit) {
- dev_warn(dev, "coherent allocation too big (requested %#x mask %#llx)\n",
- size, mask);
- return NULL;
- }
-#endif
-
- if (!mask)
- return NULL;
-
- if (mask < 0xffffffffULL)
- gfp |= GFP_DMA;
page = alloc_pages(gfp, order);
if (!page)
for (p = page + (size >> PAGE_SHIFT), e = page + (1 << order); p < e; p++)
__free_page(p);
- /*
- * Ensure that the allocated pages are zeroed, and that any data
- * lurking in the kernel direct-mapped region is invalidated.
- */
- ptr = page_address(page);
- memset(ptr, 0, size);
- dmac_flush_range(ptr, ptr + size);
- outer_flush_range(__pa(ptr), __pa(ptr) + size);
+ __dma_clear_buffer(page, size);
return page;
}
}
#ifdef CONFIG_MMU
+#ifdef CONFIG_HUGETLB_PAGE
+#error ARM Coherent DMA allocator does not (yet) support huge TLB
+#endif
-#define CONSISTENT_OFFSET(x) (((unsigned long)(x) - consistent_base) >> PAGE_SHIFT)
-#define CONSISTENT_PTE_INDEX(x) (((unsigned long)(x) - consistent_base) >> PMD_SHIFT)
-
-/*
- * These are the page tables (2MB each) covering uncached, DMA consistent allocations
- */
-static pte_t **consistent_pte;
-
-#define DEFAULT_CONSISTENT_DMA_SIZE SZ_2M
+static void *__alloc_from_contiguous(struct device *dev, size_t size,
+ pgprot_t prot, struct page **ret_page);
-unsigned long consistent_base = CONSISTENT_END - DEFAULT_CONSISTENT_DMA_SIZE;
+static void *__alloc_remap_buffer(struct device *dev, size_t size, gfp_t gfp,
+ pgprot_t prot, struct page **ret_page,
+ const void *caller);
-void __init init_consistent_dma_size(unsigned long size)
+static void *
+__dma_alloc_remap(struct page *page, size_t size, gfp_t gfp, pgprot_t prot,
+ const void *caller)
{
- unsigned long base = CONSISTENT_END - ALIGN(size, SZ_2M);
+ struct vm_struct *area;
+ unsigned long addr;
+
+ area = get_vm_area_caller(size, VM_DMA | VM_USERMAP, caller);
+ if (!area)
+ return NULL;
+ addr = (unsigned long)area->addr;
+ area->phys_addr = __pfn_to_phys(page_to_pfn(page));
- BUG_ON(consistent_pte); /* Check we're called before DMA region init */
- BUG_ON(base < VMALLOC_END);
+ if (ioremap_page_range(addr, addr + size, area->phys_addr, prot)) {
+ vunmap((void *)addr);
+ return NULL;
+ }
+ return (void *)addr;
+}
- /* Grow region to accommodate specified size */
- if (base < consistent_base)
- consistent_base = base;
+static void __dma_free_remap(void *cpu_addr, size_t size)
+{
+ struct vm_struct *area = find_vm_area(cpu_addr);
+ if (!area || !(area->flags & VM_DMA)) {
+ pr_err("%s: trying to free invalid coherent area: %p\n",
+ __func__, cpu_addr);
+ dump_stack();
+ return;
+ }
+ unmap_kernel_range((unsigned long)cpu_addr, size);
+ vunmap(cpu_addr);
}
-#include "vmregion.h"
+struct dma_pool {
+ size_t size;
+ spinlock_t lock;
+ unsigned long *bitmap;
+ unsigned long nr_pages;
+ void *vaddr;
+ struct page *page;
+};
-static struct arm_vmregion_head consistent_head = {
- .vm_lock = __SPIN_LOCK_UNLOCKED(&consistent_head.vm_lock),
- .vm_list = LIST_HEAD_INIT(consistent_head.vm_list),
- .vm_end = CONSISTENT_END,
+static struct dma_pool atomic_pool = {
+ .size = SZ_256K,
};
-#ifdef CONFIG_HUGETLB_PAGE
-#error ARM Coherent DMA allocator does not (yet) support huge TLB
-#endif
+static int __init early_coherent_pool(char *p)
+{
+ atomic_pool.size = memparse(p, &p);
+ return 0;
+}
+early_param("coherent_pool", early_coherent_pool);
/*
- * Initialise the consistent memory allocation.
+ * Initialise the coherent pool for atomic allocations.
*/
-static int __init consistent_init(void)
+static int __init atomic_pool_init(void)
{
- int ret = 0;
- pgd_t *pgd;
- pud_t *pud;
- pmd_t *pmd;
- pte_t *pte;
- int i = 0;
- unsigned long base = consistent_base;
- unsigned long num_ptes = (CONSISTENT_END - base) >> PMD_SHIFT;
+ struct dma_pool *pool = &atomic_pool;
+ pgprot_t prot = pgprot_dmacoherent(pgprot_kernel);
+ unsigned long nr_pages = pool->size >> PAGE_SHIFT;
+ unsigned long *bitmap;
+ struct page *page;
+ void *ptr;
+ int bitmap_size = BITS_TO_LONGS(nr_pages) * sizeof(long);
- consistent_pte = kmalloc(num_ptes * sizeof(pte_t), GFP_KERNEL);
- if (!consistent_pte) {
- pr_err("%s: no memory\n", __func__);
- return -ENOMEM;
+ bitmap = kzalloc(bitmap_size, GFP_KERNEL);
+ if (!bitmap)
+ goto no_bitmap;
+
+ if (IS_ENABLED(CONFIG_CMA))
+ ptr = __alloc_from_contiguous(NULL, pool->size, prot, &page);
+ else
+ ptr = __alloc_remap_buffer(NULL, pool->size, GFP_KERNEL, prot,
+ &page, NULL);
+ if (ptr) {
+ spin_lock_init(&pool->lock);
+ pool->vaddr = ptr;
+ pool->page = page;
+ pool->bitmap = bitmap;
+ pool->nr_pages = nr_pages;
+ pr_info("DMA: preallocated %u KiB pool for atomic coherent allocations\n",
+ (unsigned)pool->size / 1024);
+ return 0;
}
+ kfree(bitmap);
+no_bitmap:
+ pr_err("DMA: failed to allocate %u KiB pool for atomic coherent allocation\n",
+ (unsigned)pool->size / 1024);
+ return -ENOMEM;
+}
+/*
+ * CMA is activated by core_initcall, so we must be called after it.
+ */
+postcore_initcall(atomic_pool_init);
- pr_debug("DMA memory: 0x%08lx - 0x%08lx:\n", base, CONSISTENT_END);
- consistent_head.vm_start = base;
+struct dma_contig_early_reserve {
+ phys_addr_t base;
+ unsigned long size;
+};
- do {
- pgd = pgd_offset(&init_mm, base);
+static struct dma_contig_early_reserve dma_mmu_remap[MAX_CMA_AREAS] __initdata;
- pud = pud_alloc(&init_mm, pgd, base);
- if (!pud) {
- printk(KERN_ERR "%s: no pud tables\n", __func__);
- ret = -ENOMEM;
- break;
- }
+static int dma_mmu_remap_num __initdata;
- pmd = pmd_alloc(&init_mm, pud, base);
- if (!pmd) {
- printk(KERN_ERR "%s: no pmd tables\n", __func__);
- ret = -ENOMEM;
- break;
- }
- WARN_ON(!pmd_none(*pmd));
+void __init dma_contiguous_early_fixup(phys_addr_t base, unsigned long size)
+{
+ dma_mmu_remap[dma_mmu_remap_num].base = base;
+ dma_mmu_remap[dma_mmu_remap_num].size = size;
+ dma_mmu_remap_num++;
+}
- pte = pte_alloc_kernel(pmd, base);
- if (!pte) {
- printk(KERN_ERR "%s: no pte tables\n", __func__);
- ret = -ENOMEM;
- break;
- }
+void __init dma_contiguous_remap(void)
+{
+ int i;
+ for (i = 0; i < dma_mmu_remap_num; i++) {
+ phys_addr_t start = dma_mmu_remap[i].base;
+ phys_addr_t end = start + dma_mmu_remap[i].size;
+ struct map_desc map;
+ unsigned long addr;
+
+ if (end > arm_lowmem_limit)
+ end = arm_lowmem_limit;
+ if (start >= end)
+ return;
+
+ map.pfn = __phys_to_pfn(start);
+ map.virtual = __phys_to_virt(start);
+ map.length = end - start;
+ map.type = MT_MEMORY_DMA_READY;
- consistent_pte[i++] = pte;
- base += PMD_SIZE;
- } while (base < CONSISTENT_END);
+ /*
+ * Clear previous low-memory mapping
+ */
+ for (addr = __phys_to_virt(start); addr < __phys_to_virt(end);
+ addr += PMD_SIZE)
+ pmd_clear(pmd_off_k(addr));
- return ret;
+ iotable_init(&map, 1);
+ }
}
-core_initcall(consistent_init);
+static int __dma_update_pte(pte_t *pte, pgtable_t token, unsigned long addr,
+ void *data)
+{
+ struct page *page = virt_to_page(addr);
+ pgprot_t prot = *(pgprot_t *)data;
-static void *
-__dma_alloc_remap(struct page *page, size_t size, gfp_t gfp, pgprot_t prot,
- const void *caller)
+ set_pte_ext(pte, mk_pte(page, prot), 0);
+ return 0;
+}
+
+static void __dma_remap(struct page *page, size_t size, pgprot_t prot)
+{
+ unsigned long start = (unsigned long) page_address(page);
+ unsigned end = start + size;
+
+ apply_to_page_range(&init_mm, start, size, __dma_update_pte, &prot);
+ dsb();
+ flush_tlb_kernel_range(start, end);
+}
+
+static void *__alloc_remap_buffer(struct device *dev, size_t size, gfp_t gfp,
+ pgprot_t prot, struct page **ret_page,
+ const void *caller)
+{
+ struct page *page;
+ void *ptr;
+ page = __dma_alloc_buffer(dev, size, gfp);
+ if (!page)
+ return NULL;
+
+ ptr = __dma_alloc_remap(page, size, gfp, prot, caller);
+ if (!ptr) {
+ __dma_free_buffer(page, size);
+ return NULL;
+ }
+
+ *ret_page = page;
+ return ptr;
+}
+
+static void *__alloc_from_pool(size_t size, struct page **ret_page)
{
- struct arm_vmregion *c;
+ struct dma_pool *pool = &atomic_pool;
+ unsigned int count = size >> PAGE_SHIFT;
+ unsigned int pageno;
+ unsigned long flags;
+ void *ptr = NULL;
size_t align;
- int bit;
- if (!consistent_pte) {
- printk(KERN_ERR "%s: not initialised\n", __func__);
+ if (!pool->vaddr) {
+ pr_err("%s: coherent pool not initialised!\n", __func__);
dump_stack();
return NULL;
}
/*
- * Align the virtual region allocation - maximum alignment is
- * a section size, minimum is a page size. This helps reduce
- * fragmentation of the DMA space, and also prevents allocations
- * smaller than a section from crossing a section boundary.
- */
- bit = fls(size - 1);
- if (bit > SECTION_SHIFT)
- bit = SECTION_SHIFT;
- align = 1 << bit;
-
- /*
- * Allocate a virtual address in the consistent mapping region.
+ * Align the region allocation - allocations from pool are rather
+ * small, so align them to their order in pages, minimum is a page
+ * size. This helps reduce fragmentation of the DMA space.
*/
- c = arm_vmregion_alloc(&consistent_head, align, size,
- gfp & ~(__GFP_DMA | __GFP_HIGHMEM), caller);
- if (c) {
- pte_t *pte;
- int idx = CONSISTENT_PTE_INDEX(c->vm_start);
- u32 off = CONSISTENT_OFFSET(c->vm_start) & (PTRS_PER_PTE-1);
-
- pte = consistent_pte[idx] + off;
- c->vm_pages = page;
-
- do {
- BUG_ON(!pte_none(*pte));
-
- set_pte_ext(pte, mk_pte(page, prot), 0);
- page++;
- pte++;
- off++;
- if (off >= PTRS_PER_PTE) {
- off = 0;
- pte = consistent_pte[++idx];
- }
- } while (size -= PAGE_SIZE);
-
- dsb();
-
- return (void *)c->vm_start;
+ align = PAGE_SIZE << get_order(size);
+
+ spin_lock_irqsave(&pool->lock, flags);
+ pageno = bitmap_find_next_zero_area(pool->bitmap, pool->nr_pages,
+ 0, count, (1 << align) - 1);
+ if (pageno < pool->nr_pages) {
+ bitmap_set(pool->bitmap, pageno, count);
+ ptr = pool->vaddr + PAGE_SIZE * pageno;
+ *ret_page = pool->page + pageno;
}
- return NULL;
+ spin_unlock_irqrestore(&pool->lock, flags);
+
+ return ptr;
}
-static void __dma_free_remap(void *cpu_addr, size_t size)
+static int __free_from_pool(void *start, size_t size)
{
- struct arm_vmregion *c;
- unsigned long addr;
- pte_t *ptep;
- int idx;
- u32 off;
+ struct dma_pool *pool = &atomic_pool;
+ unsigned long pageno, count;
+ unsigned long flags;
- c = arm_vmregion_find_remove(&consistent_head, (unsigned long)cpu_addr);
- if (!c) {
- printk(KERN_ERR "%s: trying to free invalid coherent area: %p\n",
- __func__, cpu_addr);
- dump_stack();
- return;
- }
+ if (start < pool->vaddr || start > pool->vaddr + pool->size)
+ return 0;
- if ((c->vm_end - c->vm_start) != size) {
- printk(KERN_ERR "%s: freeing wrong coherent size (%ld != %d)\n",
- __func__, c->vm_end - c->vm_start, size);
+ if (start + size > pool->vaddr + pool->size) {
+ pr_err("%s: freeing wrong coherent size from pool\n", __func__);
dump_stack();
- size = c->vm_end - c->vm_start;
+ return 0;
}
- idx = CONSISTENT_PTE_INDEX(c->vm_start);
- off = CONSISTENT_OFFSET(c->vm_start) & (PTRS_PER_PTE-1);
- ptep = consistent_pte[idx] + off;
- addr = c->vm_start;
- do {
- pte_t pte = ptep_get_and_clear(&init_mm, addr, ptep);
-
- ptep++;
- addr += PAGE_SIZE;
- off++;
- if (off >= PTRS_PER_PTE) {
- off = 0;
- ptep = consistent_pte[++idx];
- }
+ pageno = (start - pool->vaddr) >> PAGE_SHIFT;
+ count = size >> PAGE_SHIFT;
- if (pte_none(pte) || !pte_present(pte))
- printk(KERN_CRIT "%s: bad page in kernel page table\n",
- __func__);
- } while (size -= PAGE_SIZE);
+ spin_lock_irqsave(&pool->lock, flags);
+ bitmap_clear(pool->bitmap, pageno, count);
+ spin_unlock_irqrestore(&pool->lock, flags);
- flush_tlb_kernel_range(c->vm_start, c->vm_end);
+ return 1;
+}
+
+static void *__alloc_from_contiguous(struct device *dev, size_t size,
+ pgprot_t prot, struct page **ret_page)
+{
+ unsigned long order = get_order(size);
+ size_t count = size >> PAGE_SHIFT;
+ struct page *page;
+
+ page = dma_alloc_from_contiguous(dev, count, order);
+ if (!page)
+ return NULL;
+
+ __dma_clear_buffer(page, size);
+ __dma_remap(page, size, prot);
+
+ *ret_page = page;
+ return page_address(page);
+}
- arm_vmregion_free(&consistent_head, c);
+static void __free_from_contiguous(struct device *dev, struct page *page,
+ size_t size)
+{
+ __dma_remap(page, size, pgprot_kernel);
+ dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT);
+}
+
+static inline pgprot_t __get_dma_pgprot(struct dma_attrs *attrs, pgprot_t prot)
+{
+ prot = dma_get_attr(DMA_ATTR_WRITE_COMBINE, attrs) ?
+ pgprot_writecombine(prot) :
+ pgprot_dmacoherent(prot);
+ return prot;
}
+#define nommu() 0
+
#else /* !CONFIG_MMU */
-#define __dma_alloc_remap(page, size, gfp, prot, c) page_address(page)
-#define __dma_free_remap(addr, size) do { } while (0)
+#define nommu() 1
+
+#define __get_dma_pgprot(attrs, prot) __pgprot(0)
+#define __alloc_remap_buffer(dev, size, gfp, prot, ret, c) NULL
+#define __alloc_from_pool(dev, size, ret_page, c) NULL
+#define __alloc_from_contiguous(dev, size, prot, ret) NULL
+#define __free_from_pool(cpu_addr, size) 0
+#define __free_from_contiguous(dev, page, size) do { } while (0)
+#define __dma_free_remap(cpu_addr, size) do { } while (0)
#endif /* CONFIG_MMU */
-static void *
-__dma_alloc(struct device *dev, size_t size, dma_addr_t *handle, gfp_t gfp,
- pgprot_t prot, const void *caller)
+static void *__alloc_simple_buffer(struct device *dev, size_t size, gfp_t gfp,
+ struct page **ret_page)
+{
+ struct page *page;
+ page = __dma_alloc_buffer(dev, size, gfp);
+ if (!page)
+ return NULL;
+
+ *ret_page = page;
+ return page_address(page);
+}
+
+
+
+static void *__dma_alloc(struct device *dev, size_t size, dma_addr_t *handle,
+ gfp_t gfp, pgprot_t prot, const void *caller)
{
+ u64 mask = get_coherent_dma_mask(dev);
struct page *page;
void *addr;
+#ifdef CONFIG_DMA_API_DEBUG
+ u64 limit = (mask + 1) & ~mask;
+ if (limit && size >= limit) {
+ dev_warn(dev, "coherent allocation too big (requested %#x mask %#llx)\n",
+ size, mask);
+ return NULL;
+ }
+#endif
+
+ if (!mask)
+ return NULL;
+
+ if (mask < 0xffffffffULL)
+ gfp |= GFP_DMA;
+
/*
* Following is a work-around (a.k.a. hack) to prevent pages
* with __GFP_COMP being passed to split_page() which cannot
*/
gfp &= ~(__GFP_COMP);
- *handle = ~0;
+ *handle = DMA_ERROR_CODE;
size = PAGE_ALIGN(size);
- page = __dma_alloc_buffer(dev, size, gfp);
- if (!page)
- return NULL;
-
- if (!arch_is_coherent())
- addr = __dma_alloc_remap(page, size, gfp, prot, caller);
+ if (arch_is_coherent() || nommu())
+ addr = __alloc_simple_buffer(dev, size, gfp, &page);
+ else if (gfp & GFP_ATOMIC)
+ addr = __alloc_from_pool(size, &page);
+ else if (!IS_ENABLED(CONFIG_CMA))
+ addr = __alloc_remap_buffer(dev, size, gfp, prot, &page, caller);
else
- addr = page_address(page);
+ addr = __alloc_from_contiguous(dev, size, prot, &page);
if (addr)
*handle = pfn_to_dma(dev, page_to_pfn(page));
- else
- __dma_free_buffer(page, size);
return addr;
}
* Allocate DMA-coherent memory space and return both the kernel remapped
* virtual and bus address for that space.
*/
-void *
-dma_alloc_coherent(struct device *dev, size_t size, dma_addr_t *handle, gfp_t gfp)
+void *arm_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle,
+ gfp_t gfp, struct dma_attrs *attrs)
{
+ pgprot_t prot = __get_dma_pgprot(attrs, pgprot_kernel);
void *memory;
if (dma_alloc_from_coherent(dev, size, handle, &memory))
return memory;
- return __dma_alloc(dev, size, handle, gfp,
- pgprot_dmacoherent(pgprot_kernel),
+ return __dma_alloc(dev, size, handle, gfp, prot,
__builtin_return_address(0));
}
-EXPORT_SYMBOL(dma_alloc_coherent);
/*
- * Allocate a writecombining region, in much the same way as
- * dma_alloc_coherent above.
+ * Create userspace mapping for the DMA-coherent memory.
*/
-void *
-dma_alloc_writecombine(struct device *dev, size_t size, dma_addr_t *handle, gfp_t gfp)
-{
- return __dma_alloc(dev, size, handle, gfp,
- pgprot_writecombine(pgprot_kernel),
- __builtin_return_address(0));
-}
-EXPORT_SYMBOL(dma_alloc_writecombine);
-
-static int dma_mmap(struct device *dev, struct vm_area_struct *vma,
- void *cpu_addr, dma_addr_t dma_addr, size_t size)
+int arm_dma_mmap(struct device *dev, struct vm_area_struct *vma,
+ void *cpu_addr, dma_addr_t dma_addr, size_t size,
+ struct dma_attrs *attrs)
{
int ret = -ENXIO;
#ifdef CONFIG_MMU
- unsigned long user_size, kern_size;
- struct arm_vmregion *c;
+ unsigned long user_count = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT;
+ unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT;
+ unsigned long pfn = dma_to_pfn(dev, dma_addr);
+ unsigned long off = vma->vm_pgoff;
- user_size = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT;
+ vma->vm_page_prot = __get_dma_pgprot(attrs, vma->vm_page_prot);
- c = arm_vmregion_find(&consistent_head, (unsigned long)cpu_addr);
- if (c) {
- unsigned long off = vma->vm_pgoff;
+ if (dma_mmap_from_coherent(dev, vma, cpu_addr, size, &ret))
+ return ret;
- kern_size = (c->vm_end - c->vm_start) >> PAGE_SHIFT;
-
- if (off < kern_size &&
- user_size <= (kern_size - off)) {
- ret = remap_pfn_range(vma, vma->vm_start,
- page_to_pfn(c->vm_pages) + off,
- user_size << PAGE_SHIFT,
- vma->vm_page_prot);
- }
+ if (off < count && user_count <= (count - off)) {
+ ret = remap_pfn_range(vma, vma->vm_start,
+ pfn + off,
+ user_count << PAGE_SHIFT,
+ vma->vm_page_prot);
}
#endif /* CONFIG_MMU */
return ret;
}
-int dma_mmap_coherent(struct device *dev, struct vm_area_struct *vma,
- void *cpu_addr, dma_addr_t dma_addr, size_t size)
-{
- vma->vm_page_prot = pgprot_dmacoherent(vma->vm_page_prot);
- return dma_mmap(dev, vma, cpu_addr, dma_addr, size);
-}
-EXPORT_SYMBOL(dma_mmap_coherent);
-
-int dma_mmap_writecombine(struct device *dev, struct vm_area_struct *vma,
- void *cpu_addr, dma_addr_t dma_addr, size_t size)
-{
- vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
- return dma_mmap(dev, vma, cpu_addr, dma_addr, size);
-}
-EXPORT_SYMBOL(dma_mmap_writecombine);
-
/*
- * free a page as defined by the above mapping.
- * Must not be called with IRQs disabled.
+ * Free a buffer as defined by the above mapping.
*/
-void dma_free_coherent(struct device *dev, size_t size, void *cpu_addr, dma_addr_t handle)
+void arm_dma_free(struct device *dev, size_t size, void *cpu_addr,
+ dma_addr_t handle, struct dma_attrs *attrs)
{
- WARN_ON(irqs_disabled());
+ struct page *page = pfn_to_page(dma_to_pfn(dev, handle));
if (dma_release_from_coherent(dev, get_order(size), cpu_addr))
return;
size = PAGE_ALIGN(size);
- if (!arch_is_coherent())
+ if (arch_is_coherent() || nommu()) {
+ __dma_free_buffer(page, size);
+ } else if (cpu_architecture() < CPU_ARCH_ARMv6) {
__dma_free_remap(cpu_addr, size);
-
- __dma_free_buffer(pfn_to_page(dma_to_pfn(dev, handle)), size);
-}
-EXPORT_SYMBOL(dma_free_coherent);
-
-/*
- * Make an area consistent for devices.
- * Note: Drivers should NOT use this function directly, as it will break
- * platforms with CONFIG_DMABOUNCE.
- * Use the driver DMA support - see dma-mapping.h (dma_sync_*)
- */
-void ___dma_single_cpu_to_dev(const void *kaddr, size_t size,
- enum dma_data_direction dir)
-{
- unsigned long paddr;
-
- BUG_ON(!virt_addr_valid(kaddr) || !virt_addr_valid(kaddr + size - 1));
-
- dmac_map_area(kaddr, size, dir);
-
- paddr = __pa(kaddr);
- if (dir == DMA_FROM_DEVICE) {
- outer_inv_range(paddr, paddr + size);
+ __dma_free_buffer(page, size);
} else {
- outer_clean_range(paddr, paddr + size);
+ if (__free_from_pool(cpu_addr, size))
+ return;
+ /*
+ * Non-atomic allocations cannot be freed with IRQs disabled
+ */
+ WARN_ON(irqs_disabled());
+ __free_from_contiguous(dev, page, size);
}
- /* FIXME: non-speculating: flush on bidirectional mappings? */
}
-EXPORT_SYMBOL(___dma_single_cpu_to_dev);
-void ___dma_single_dev_to_cpu(const void *kaddr, size_t size,
- enum dma_data_direction dir)
+int arm_dma_get_sgtable(struct device *dev, struct sg_table *sgt,
+ void *cpu_addr, dma_addr_t handle, size_t size,
+ struct dma_attrs *attrs)
{
- BUG_ON(!virt_addr_valid(kaddr) || !virt_addr_valid(kaddr + size - 1));
+ struct page *page = pfn_to_page(dma_to_pfn(dev, handle));
+ int ret;
- /* FIXME: non-speculating: not required */
- /* don't bother invalidating if DMA to device */
- if (dir != DMA_TO_DEVICE) {
- unsigned long paddr = __pa(kaddr);
- outer_inv_range(paddr, paddr + size);
- }
+ ret = sg_alloc_table(sgt, 1, GFP_KERNEL);
+ if (unlikely(ret))
+ return ret;
- dmac_unmap_area(kaddr, size, dir);
+ sg_set_page(sgt->sgl, page, PAGE_ALIGN(size), 0);
+ return 0;
}
-EXPORT_SYMBOL(___dma_single_dev_to_cpu);
static void dma_cache_maint_page(struct page *page, unsigned long offset,
size_t size, enum dma_data_direction dir,
} while (left);
}
-void ___dma_page_cpu_to_dev(struct page *page, unsigned long off,
- size_t size, enum dma_data_direction dir)
-{
+/*
+ * Make an area consistent for devices.
+ * Note: Drivers should NOT use this function directly, as it will break
+ * platforms with CONFIG_DMABOUNCE.
+ * Use the driver DMA support - see dma-mapping.h (dma_sync_*)
+ */
+static void __dma_page_cpu_to_dev(struct page *page, unsigned long off,
+ size_t size, enum dma_data_direction dir)
+{
unsigned long paddr;
dma_cache_maint_page(page, off, size, dir, dmac_map_area);
}
/* FIXME: non-speculating: flush on bidirectional mappings? */
}
-EXPORT_SYMBOL(___dma_page_cpu_to_dev);
-void ___dma_page_dev_to_cpu(struct page *page, unsigned long off,
+static void __dma_page_dev_to_cpu(struct page *page, unsigned long off,
size_t size, enum dma_data_direction dir)
{
unsigned long paddr = page_to_phys(page) + off;
if (dir != DMA_TO_DEVICE && off == 0 && size >= PAGE_SIZE)
set_bit(PG_dcache_clean, &page->flags);
}
-EXPORT_SYMBOL(___dma_page_dev_to_cpu);
/**
- * dma_map_sg - map a set of SG buffers for streaming mode DMA
+ * arm_dma_map_sg - map a set of SG buffers for streaming mode DMA
* @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
* @sg: list of buffers
* @nents: number of buffers to map
* Device ownership issues as mentioned for dma_map_single are the same
* here.
*/
-int dma_map_sg(struct device *dev, struct scatterlist *sg, int nents,
- enum dma_data_direction dir)
+int arm_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents,
+ enum dma_data_direction dir, struct dma_attrs *attrs)
{
+ struct dma_map_ops *ops = get_dma_ops(dev);
struct scatterlist *s;
int i, j;
- BUG_ON(!valid_dma_direction(dir));
-
for_each_sg(sg, s, nents, i) {
- s->dma_address = __dma_map_page(dev, sg_page(s), s->offset,
- s->length, dir);
+#ifdef CONFIG_NEED_SG_DMA_LENGTH
+ s->dma_length = s->length;
+#endif
+ s->dma_address = ops->map_page(dev, sg_page(s), s->offset,
+ s->length, dir, attrs);
if (dma_mapping_error(dev, s->dma_address))
goto bad_mapping;
}
- debug_dma_map_sg(dev, sg, nents, nents, dir);
return nents;
bad_mapping:
for_each_sg(sg, s, i, j)
- __dma_unmap_page(dev, sg_dma_address(s), sg_dma_len(s), dir);
+ ops->unmap_page(dev, sg_dma_address(s), sg_dma_len(s), dir, attrs);
return 0;
}
-EXPORT_SYMBOL(dma_map_sg);
/**
- * dma_unmap_sg - unmap a set of SG buffers mapped by dma_map_sg
+ * arm_dma_unmap_sg - unmap a set of SG buffers mapped by dma_map_sg
* @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
* @sg: list of buffers
* @nents: number of buffers to unmap (same as was passed to dma_map_sg)
* Unmap a set of streaming mode DMA translations. Again, CPU access
* rules concerning calls here are the same as for dma_unmap_single().
*/
-void dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,
- enum dma_data_direction dir)
+void arm_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,
+ enum dma_data_direction dir, struct dma_attrs *attrs)
{
+ struct dma_map_ops *ops = get_dma_ops(dev);
struct scatterlist *s;
- int i;
- debug_dma_unmap_sg(dev, sg, nents, dir);
+ int i;
for_each_sg(sg, s, nents, i)
- __dma_unmap_page(dev, sg_dma_address(s), sg_dma_len(s), dir);
+ ops->unmap_page(dev, sg_dma_address(s), sg_dma_len(s), dir, attrs);
}
-EXPORT_SYMBOL(dma_unmap_sg);
/**
- * dma_sync_sg_for_cpu
+ * arm_dma_sync_sg_for_cpu
* @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
* @sg: list of buffers
* @nents: number of buffers to map (returned from dma_map_sg)
* @dir: DMA transfer direction (same as was passed to dma_map_sg)
*/
-void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg,
+void arm_dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg,
int nents, enum dma_data_direction dir)
{
+ struct dma_map_ops *ops = get_dma_ops(dev);
struct scatterlist *s;
int i;
- for_each_sg(sg, s, nents, i) {
- if (!dmabounce_sync_for_cpu(dev, sg_dma_address(s), 0,
- sg_dma_len(s), dir))
- continue;
-
- __dma_page_dev_to_cpu(sg_page(s), s->offset,
- s->length, dir);
- }
-
- debug_dma_sync_sg_for_cpu(dev, sg, nents, dir);
+ for_each_sg(sg, s, nents, i)
+ ops->sync_single_for_cpu(dev, sg_dma_address(s), s->length,
+ dir);
}
-EXPORT_SYMBOL(dma_sync_sg_for_cpu);
/**
- * dma_sync_sg_for_device
+ * arm_dma_sync_sg_for_device
* @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
* @sg: list of buffers
* @nents: number of buffers to map (returned from dma_map_sg)
* @dir: DMA transfer direction (same as was passed to dma_map_sg)
*/
-void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg,
+void arm_dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg,
int nents, enum dma_data_direction dir)
{
+ struct dma_map_ops *ops = get_dma_ops(dev);
struct scatterlist *s;
int i;
- for_each_sg(sg, s, nents, i) {
- if (!dmabounce_sync_for_device(dev, sg_dma_address(s), 0,
- sg_dma_len(s), dir))
- continue;
-
- __dma_page_cpu_to_dev(sg_page(s), s->offset,
- s->length, dir);
- }
-
- debug_dma_sync_sg_for_device(dev, sg, nents, dir);
+ for_each_sg(sg, s, nents, i)
+ ops->sync_single_for_device(dev, sg_dma_address(s), s->length,
+ dir);
}
-EXPORT_SYMBOL(dma_sync_sg_for_device);
/*
* Return whether the given device DMA address mask can be supported
}
EXPORT_SYMBOL(dma_supported);
-int dma_set_mask(struct device *dev, u64 dma_mask)
+static int arm_dma_set_mask(struct device *dev, u64 dma_mask)
{
if (!dev->dma_mask || !dma_supported(dev, dma_mask))
return -EIO;
-#ifndef CONFIG_DMABOUNCE
*dev->dma_mask = dma_mask;
-#endif
return 0;
}
-EXPORT_SYMBOL(dma_set_mask);
#define PREALLOC_DMA_DEBUG_ENTRIES 4096
static int __init dma_debug_do_init(void)
{
-#ifdef CONFIG_MMU
- arm_vmregion_create_proc("dma-mappings", &consistent_head);
-#endif
dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES);
return 0;
}
fs_initcall(dma_debug_do_init);
+
+#ifdef CONFIG_ARM_DMA_USE_IOMMU
+
+/* IOMMU */
+
+static inline dma_addr_t __alloc_iova(struct dma_iommu_mapping *mapping,
+ size_t size)
+{
+ unsigned int order = get_order(size);
+ unsigned int align = 0;
+ unsigned int count, start;
+ unsigned long flags;
+
+ count = ((PAGE_ALIGN(size) >> PAGE_SHIFT) +
+ (1 << mapping->order) - 1) >> mapping->order;
+
+ if (order > mapping->order)
+ align = (1 << (order - mapping->order)) - 1;
+
+ spin_lock_irqsave(&mapping->lock, flags);
+ start = bitmap_find_next_zero_area(mapping->bitmap, mapping->bits, 0,
+ count, align);
+ if (start > mapping->bits) {
+ spin_unlock_irqrestore(&mapping->lock, flags);
+ return DMA_ERROR_CODE;
+ }
+
+ bitmap_set(mapping->bitmap, start, count);
+ spin_unlock_irqrestore(&mapping->lock, flags);
+
+ return mapping->base + (start << (mapping->order + PAGE_SHIFT));
+}
+
+static inline void __free_iova(struct dma_iommu_mapping *mapping,
+ dma_addr_t addr, size_t size)
+{
+ unsigned int start = (addr - mapping->base) >>
+ (mapping->order + PAGE_SHIFT);
+ unsigned int count = ((size >> PAGE_SHIFT) +
+ (1 << mapping->order) - 1) >> mapping->order;
+ unsigned long flags;
+
+ spin_lock_irqsave(&mapping->lock, flags);
+ bitmap_clear(mapping->bitmap, start, count);
+ spin_unlock_irqrestore(&mapping->lock, flags);
+}
+
+static struct page **__iommu_alloc_buffer(struct device *dev, size_t size, gfp_t gfp)
+{
+ struct page **pages;
+ int count = size >> PAGE_SHIFT;
+ int array_size = count * sizeof(struct page *);
+ int i = 0;
+
+ if (array_size <= PAGE_SIZE)
+ pages = kzalloc(array_size, gfp);
+ else
+ pages = vzalloc(array_size);
+ if (!pages)
+ return NULL;
+
+ while (count) {
+ int j, order = __ffs(count);
+
+ pages[i] = alloc_pages(gfp | __GFP_NOWARN, order);
+ while (!pages[i] && order)
+ pages[i] = alloc_pages(gfp | __GFP_NOWARN, --order);
+ if (!pages[i])
+ goto error;
+
+ if (order)
+ split_page(pages[i], order);
+ j = 1 << order;
+ while (--j)
+ pages[i + j] = pages[i] + j;
+
+ __dma_clear_buffer(pages[i], PAGE_SIZE << order);
+ i += 1 << order;
+ count -= 1 << order;
+ }
+
+ return pages;
+error:
+ while (--i)
+ if (pages[i])
+ __free_pages(pages[i], 0);
+ if (array_size < PAGE_SIZE)
+ kfree(pages);
+ else
+ vfree(pages);
+ return NULL;
+}
+
+static int __iommu_free_buffer(struct device *dev, struct page **pages, size_t size)
+{
+ int count = size >> PAGE_SHIFT;
+ int array_size = count * sizeof(struct page *);
+ int i;
+ for (i = 0; i < count; i++)
+ if (pages[i])
+ __free_pages(pages[i], 0);
+ if (array_size < PAGE_SIZE)
+ kfree(pages);
+ else
+ vfree(pages);
+ return 0;
+}
+
+/*
+ * Create a CPU mapping for a specified pages
+ */
+static void *
+__iommu_alloc_remap(struct page **pages, size_t size, gfp_t gfp, pgprot_t prot,
+ const void *caller)
+{
+ unsigned int i, nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT;
+ struct vm_struct *area;
+ unsigned long p;
+
+ area = get_vm_area_caller(size, VM_DMA | VM_USERMAP, caller);
+ if (!area)
+ return NULL;
+
+ area->pages = pages;
+ area->nr_pages = nr_pages;
+ p = (unsigned long)area->addr;
+
+ for (i = 0; i < nr_pages; i++) {
+ phys_addr_t phys = __pfn_to_phys(page_to_pfn(pages[i]));
+ if (ioremap_page_range(p, p + PAGE_SIZE, phys, prot))
+ goto err;
+ p += PAGE_SIZE;
+ }
+ return area->addr;
+err:
+ unmap_kernel_range((unsigned long)area->addr, size);
+ vunmap(area->addr);
+ return NULL;
+}
+
+/*
+ * Create a mapping in device IO address space for specified pages
+ */
+static dma_addr_t
+__iommu_create_mapping(struct device *dev, struct page **pages, size_t size)
+{
+ struct dma_iommu_mapping *mapping = dev->archdata.mapping;
+ unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT;
+ dma_addr_t dma_addr, iova;
+ int i, ret = DMA_ERROR_CODE;
+
+ dma_addr = __alloc_iova(mapping, size);
+ if (dma_addr == DMA_ERROR_CODE)
+ return dma_addr;
+
+ iova = dma_addr;
+ for (i = 0; i < count; ) {
+ unsigned int next_pfn = page_to_pfn(pages[i]) + 1;
+ phys_addr_t phys = page_to_phys(pages[i]);
+ unsigned int len, j;
+
+ for (j = i + 1; j < count; j++, next_pfn++)
+ if (page_to_pfn(pages[j]) != next_pfn)
+ break;
+
+ len = (j - i) << PAGE_SHIFT;
+ ret = iommu_map(mapping->domain, iova, phys, len, 0);
+ if (ret < 0)
+ goto fail;
+ iova += len;
+ i = j;
+ }
+ return dma_addr;
+fail:
+ iommu_unmap(mapping->domain, dma_addr, iova-dma_addr);
+ __free_iova(mapping, dma_addr, size);
+ return DMA_ERROR_CODE;
+}
+
+static int __iommu_remove_mapping(struct device *dev, dma_addr_t iova, size_t size)
+{
+ struct dma_iommu_mapping *mapping = dev->archdata.mapping;
+
+ /*
+ * add optional in-page offset from iova to size and align
+ * result to page size
+ */
+ size = PAGE_ALIGN((iova & ~PAGE_MASK) + size);
+ iova &= PAGE_MASK;
+
+ iommu_unmap(mapping->domain, iova, size);
+ __free_iova(mapping, iova, size);
+ return 0;
+}
+
+static struct page **__iommu_get_pages(void *cpu_addr, struct dma_attrs *attrs)
+{
+ struct vm_struct *area;
+
+ if (dma_get_attr(DMA_ATTR_NO_KERNEL_MAPPING, attrs))
+ return cpu_addr;
+
+ area = find_vm_area(cpu_addr);
+ if (area && (area->flags & VM_DMA))
+ return area->pages;
+ return NULL;
+}
+
+static void *arm_iommu_alloc_attrs(struct device *dev, size_t size,
+ dma_addr_t *handle, gfp_t gfp, struct dma_attrs *attrs)
+{
+ pgprot_t prot = __get_dma_pgprot(attrs, pgprot_kernel);
+ struct page **pages;
+ void *addr = NULL;
+
+ *handle = DMA_ERROR_CODE;
+ size = PAGE_ALIGN(size);
+
+ pages = __iommu_alloc_buffer(dev, size, gfp);
+ if (!pages)
+ return NULL;
+
+ *handle = __iommu_create_mapping(dev, pages, size);
+ if (*handle == DMA_ERROR_CODE)
+ goto err_buffer;
+
+ if (dma_get_attr(DMA_ATTR_NO_KERNEL_MAPPING, attrs))
+ return pages;
+
+ addr = __iommu_alloc_remap(pages, size, gfp, prot,
+ __builtin_return_address(0));
+ if (!addr)
+ goto err_mapping;
+
+ return addr;
+
+err_mapping:
+ __iommu_remove_mapping(dev, *handle, size);
+err_buffer:
+ __iommu_free_buffer(dev, pages, size);
+ return NULL;
+}
+
+static int arm_iommu_mmap_attrs(struct device *dev, struct vm_area_struct *vma,
+ void *cpu_addr, dma_addr_t dma_addr, size_t size,
+ struct dma_attrs *attrs)
+{
+ unsigned long uaddr = vma->vm_start;
+ unsigned long usize = vma->vm_end - vma->vm_start;
+ struct page **pages = __iommu_get_pages(cpu_addr, attrs);
+
+ vma->vm_page_prot = __get_dma_pgprot(attrs, vma->vm_page_prot);
+
+ if (!pages)
+ return -ENXIO;
+
+ do {
+ int ret = vm_insert_page(vma, uaddr, *pages++);
+ if (ret) {
+ pr_err("Remapping memory failed: %d\n", ret);
+ return ret;
+ }
+ uaddr += PAGE_SIZE;
+ usize -= PAGE_SIZE;
+ } while (usize > 0);
+
+ return 0;
+}
+
+/*
+ * free a page as defined by the above mapping.
+ * Must not be called with IRQs disabled.
+ */
+void arm_iommu_free_attrs(struct device *dev, size_t size, void *cpu_addr,
+ dma_addr_t handle, struct dma_attrs *attrs)
+{
+ struct page **pages = __iommu_get_pages(cpu_addr, attrs);
+ size = PAGE_ALIGN(size);
+
+ if (!pages) {
+ pr_err("%s: trying to free invalid coherent area: %p\n",
+ __func__, cpu_addr);
+ dump_stack();
+ return;
+ }
+
+ if (!dma_get_attr(DMA_ATTR_NO_KERNEL_MAPPING, attrs)) {
+ unmap_kernel_range((unsigned long)cpu_addr, size);
+ vunmap(cpu_addr);
+ }
+
+ __iommu_remove_mapping(dev, handle, size);
+ __iommu_free_buffer(dev, pages, size);
+}
+
+static int arm_iommu_get_sgtable(struct device *dev, struct sg_table *sgt,
+ void *cpu_addr, dma_addr_t dma_addr,
+ size_t size, struct dma_attrs *attrs)
+{
+ unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT;
+ struct page **pages = __iommu_get_pages(cpu_addr, attrs);
+
+ if (!pages)
+ return -ENXIO;
+
+ return sg_alloc_table_from_pages(sgt, pages, count, 0, size,
+ GFP_KERNEL);
+}
+
+/*
+ * Map a part of the scatter-gather list into contiguous io address space
+ */
+static int __map_sg_chunk(struct device *dev, struct scatterlist *sg,
+ size_t size, dma_addr_t *handle,
+ enum dma_data_direction dir, struct dma_attrs *attrs)
+{
+ struct dma_iommu_mapping *mapping = dev->archdata.mapping;
+ dma_addr_t iova, iova_base;
+ int ret = 0;
+ unsigned int count;
+ struct scatterlist *s;
+
+ size = PAGE_ALIGN(size);
+ *handle = DMA_ERROR_CODE;
+
+ iova_base = iova = __alloc_iova(mapping, size);
+ if (iova == DMA_ERROR_CODE)
+ return -ENOMEM;
+
+ for (count = 0, s = sg; count < (size >> PAGE_SHIFT); s = sg_next(s)) {
+ phys_addr_t phys = page_to_phys(sg_page(s));
+ unsigned int len = PAGE_ALIGN(s->offset + s->length);
+
+ if (!arch_is_coherent() &&
+ !dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs))
+ __dma_page_cpu_to_dev(sg_page(s), s->offset, s->length, dir);
+
+ ret = iommu_map(mapping->domain, iova, phys, len, 0);
+ if (ret < 0)
+ goto fail;
+ count += len >> PAGE_SHIFT;
+ iova += len;
+ }
+ *handle = iova_base;
+
+ return 0;
+fail:
+ iommu_unmap(mapping->domain, iova_base, count * PAGE_SIZE);
+ __free_iova(mapping, iova_base, size);
+ return ret;
+}
+
+/**
+ * arm_iommu_map_sg - map a set of SG buffers for streaming mode DMA
+ * @dev: valid struct device pointer
+ * @sg: list of buffers
+ * @nents: number of buffers to map
+ * @dir: DMA transfer direction
+ *
+ * Map a set of buffers described by scatterlist in streaming mode for DMA.
+ * The scatter gather list elements are merged together (if possible) and
+ * tagged with the appropriate dma address and length. They are obtained via
+ * sg_dma_{address,length}.
+ */
+int arm_iommu_map_sg(struct device *dev, struct scatterlist *sg, int nents,
+ enum dma_data_direction dir, struct dma_attrs *attrs)
+{
+ struct scatterlist *s = sg, *dma = sg, *start = sg;
+ int i, count = 0;
+ unsigned int offset = s->offset;
+ unsigned int size = s->offset + s->length;
+ unsigned int max = dma_get_max_seg_size(dev);
+
+ for (i = 1; i < nents; i++) {
+ s = sg_next(s);
+
+ s->dma_address = DMA_ERROR_CODE;
+ s->dma_length = 0;
+
+ if (s->offset || (size & ~PAGE_MASK) || size + s->length > max) {
+ if (__map_sg_chunk(dev, start, size, &dma->dma_address,
+ dir, attrs) < 0)
+ goto bad_mapping;
+
+ dma->dma_address += offset;
+ dma->dma_length = size - offset;
+
+ size = offset = s->offset;
+ start = s;
+ dma = sg_next(dma);
+ count += 1;
+ }
+ size += s->length;
+ }
+ if (__map_sg_chunk(dev, start, size, &dma->dma_address, dir, attrs) < 0)
+ goto bad_mapping;
+
+ dma->dma_address += offset;
+ dma->dma_length = size - offset;
+
+ return count+1;
+
+bad_mapping:
+ for_each_sg(sg, s, count, i)
+ __iommu_remove_mapping(dev, sg_dma_address(s), sg_dma_len(s));
+ return 0;
+}
+
+/**
+ * arm_iommu_unmap_sg - unmap a set of SG buffers mapped by dma_map_sg
+ * @dev: valid struct device pointer
+ * @sg: list of buffers
+ * @nents: number of buffers to unmap (same as was passed to dma_map_sg)
+ * @dir: DMA transfer direction (same as was passed to dma_map_sg)
+ *
+ * Unmap a set of streaming mode DMA translations. Again, CPU access
+ * rules concerning calls here are the same as for dma_unmap_single().
+ */
+void arm_iommu_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,
+ enum dma_data_direction dir, struct dma_attrs *attrs)
+{
+ struct scatterlist *s;
+ int i;
+
+ for_each_sg(sg, s, nents, i) {
+ if (sg_dma_len(s))
+ __iommu_remove_mapping(dev, sg_dma_address(s),
+ sg_dma_len(s));
+ if (!arch_is_coherent() &&
+ !dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs))
+ __dma_page_dev_to_cpu(sg_page(s), s->offset,
+ s->length, dir);
+ }
+}
+
+/**
+ * arm_iommu_sync_sg_for_cpu
+ * @dev: valid struct device pointer
+ * @sg: list of buffers
+ * @nents: number of buffers to map (returned from dma_map_sg)
+ * @dir: DMA transfer direction (same as was passed to dma_map_sg)
+ */
+void arm_iommu_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg,
+ int nents, enum dma_data_direction dir)
+{
+ struct scatterlist *s;
+ int i;
+
+ for_each_sg(sg, s, nents, i)
+ if (!arch_is_coherent())
+ __dma_page_dev_to_cpu(sg_page(s), s->offset, s->length, dir);
+
+}
+
+/**
+ * arm_iommu_sync_sg_for_device
+ * @dev: valid struct device pointer
+ * @sg: list of buffers
+ * @nents: number of buffers to map (returned from dma_map_sg)
+ * @dir: DMA transfer direction (same as was passed to dma_map_sg)
+ */
+void arm_iommu_sync_sg_for_device(struct device *dev, struct scatterlist *sg,
+ int nents, enum dma_data_direction dir)
+{
+ struct scatterlist *s;
+ int i;
+
+ for_each_sg(sg, s, nents, i)
+ if (!arch_is_coherent())
+ __dma_page_cpu_to_dev(sg_page(s), s->offset, s->length, dir);
+}
+
+
+/**
+ * arm_iommu_map_page
+ * @dev: valid struct device pointer
+ * @page: page that buffer resides in
+ * @offset: offset into page for start of buffer
+ * @size: size of buffer to map
+ * @dir: DMA transfer direction
+ *
+ * IOMMU aware version of arm_dma_map_page()
+ */
+static dma_addr_t arm_iommu_map_page(struct device *dev, struct page *page,
+ unsigned long offset, size_t size, enum dma_data_direction dir,
+ struct dma_attrs *attrs)
+{
+ struct dma_iommu_mapping *mapping = dev->archdata.mapping;
+ dma_addr_t dma_addr;
+ int ret, len = PAGE_ALIGN(size + offset);
+
+ if (!arch_is_coherent() && !dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs))
+ __dma_page_cpu_to_dev(page, offset, size, dir);
+
+ dma_addr = __alloc_iova(mapping, len);
+ if (dma_addr == DMA_ERROR_CODE)
+ return dma_addr;
+
+ ret = iommu_map(mapping->domain, dma_addr, page_to_phys(page), len, 0);
+ if (ret < 0)
+ goto fail;
+
+ return dma_addr + offset;
+fail:
+ __free_iova(mapping, dma_addr, len);
+ return DMA_ERROR_CODE;
+}
+
+/**
+ * arm_iommu_unmap_page
+ * @dev: valid struct device pointer
+ * @handle: DMA address of buffer
+ * @size: size of buffer (same as passed to dma_map_page)
+ * @dir: DMA transfer direction (same as passed to dma_map_page)
+ *
+ * IOMMU aware version of arm_dma_unmap_page()
+ */
+static void arm_iommu_unmap_page(struct device *dev, dma_addr_t handle,
+ size_t size, enum dma_data_direction dir,
+ struct dma_attrs *attrs)
+{
+ struct dma_iommu_mapping *mapping = dev->archdata.mapping;
+ dma_addr_t iova = handle & PAGE_MASK;
+ struct page *page = phys_to_page(iommu_iova_to_phys(mapping->domain, iova));
+ int offset = handle & ~PAGE_MASK;
+ int len = PAGE_ALIGN(size + offset);
+
+ if (!iova)
+ return;
+
+ if (!arch_is_coherent() && !dma_get_attr(DMA_ATTR_SKIP_CPU_SYNC, attrs))
+ __dma_page_dev_to_cpu(page, offset, size, dir);
+
+ iommu_unmap(mapping->domain, iova, len);
+ __free_iova(mapping, iova, len);
+}
+
+static void arm_iommu_sync_single_for_cpu(struct device *dev,
+ dma_addr_t handle, size_t size, enum dma_data_direction dir)
+{
+ struct dma_iommu_mapping *mapping = dev->archdata.mapping;
+ dma_addr_t iova = handle & PAGE_MASK;
+ struct page *page = phys_to_page(iommu_iova_to_phys(mapping->domain, iova));
+ unsigned int offset = handle & ~PAGE_MASK;
+
+ if (!iova)
+ return;
+
+ if (!arch_is_coherent())
+ __dma_page_dev_to_cpu(page, offset, size, dir);
+}
+
+static void arm_iommu_sync_single_for_device(struct device *dev,
+ dma_addr_t handle, size_t size, enum dma_data_direction dir)
+{
+ struct dma_iommu_mapping *mapping = dev->archdata.mapping;
+ dma_addr_t iova = handle & PAGE_MASK;
+ struct page *page = phys_to_page(iommu_iova_to_phys(mapping->domain, iova));
+ unsigned int offset = handle & ~PAGE_MASK;
+
+ if (!iova)
+ return;
+
+ __dma_page_cpu_to_dev(page, offset, size, dir);
+}
+
+struct dma_map_ops iommu_ops = {
+ .alloc = arm_iommu_alloc_attrs,
+ .free = arm_iommu_free_attrs,
+ .mmap = arm_iommu_mmap_attrs,
+ .get_sgtable = arm_iommu_get_sgtable,
+
+ .map_page = arm_iommu_map_page,
+ .unmap_page = arm_iommu_unmap_page,
+ .sync_single_for_cpu = arm_iommu_sync_single_for_cpu,
+ .sync_single_for_device = arm_iommu_sync_single_for_device,
+
+ .map_sg = arm_iommu_map_sg,
+ .unmap_sg = arm_iommu_unmap_sg,
+ .sync_sg_for_cpu = arm_iommu_sync_sg_for_cpu,
+ .sync_sg_for_device = arm_iommu_sync_sg_for_device,
+};
+
+/**
+ * arm_iommu_create_mapping
+ * @bus: pointer to the bus holding the client device (for IOMMU calls)
+ * @base: start address of the valid IO address space
+ * @size: size of the valid IO address space
+ * @order: accuracy of the IO addresses allocations
+ *
+ * Creates a mapping structure which holds information about used/unused
+ * IO address ranges, which is required to perform memory allocation and
+ * mapping with IOMMU aware functions.
+ *
+ * The client device need to be attached to the mapping with
+ * arm_iommu_attach_device function.
+ */
+struct dma_iommu_mapping *
+arm_iommu_create_mapping(struct bus_type *bus, dma_addr_t base, size_t size,
+ int order)
+{
+ unsigned int count = size >> (PAGE_SHIFT + order);
+ unsigned int bitmap_size = BITS_TO_LONGS(count) * sizeof(long);
+ struct dma_iommu_mapping *mapping;
+ int err = -ENOMEM;
+
+ if (!count)
+ return ERR_PTR(-EINVAL);
+
+ mapping = kzalloc(sizeof(struct dma_iommu_mapping), GFP_KERNEL);
+ if (!mapping)
+ goto err;
+
+ mapping->bitmap = kzalloc(bitmap_size, GFP_KERNEL);
+ if (!mapping->bitmap)
+ goto err2;
+
+ mapping->base = base;
+ mapping->bits = BITS_PER_BYTE * bitmap_size;
+ mapping->order = order;
+ spin_lock_init(&mapping->lock);
+
+ mapping->domain = iommu_domain_alloc(bus);
+ if (!mapping->domain)
+ goto err3;
+
+ kref_init(&mapping->kref);
+ return mapping;
+err3:
+ kfree(mapping->bitmap);
+err2:
+ kfree(mapping);
+err:
+ return ERR_PTR(err);
+}
+
+static void release_iommu_mapping(struct kref *kref)
+{
+ struct dma_iommu_mapping *mapping =
+ container_of(kref, struct dma_iommu_mapping, kref);
+
+ iommu_domain_free(mapping->domain);
+ kfree(mapping->bitmap);
+ kfree(mapping);
+}
+
+void arm_iommu_release_mapping(struct dma_iommu_mapping *mapping)
+{
+ if (mapping)
+ kref_put(&mapping->kref, release_iommu_mapping);
+}
+
+/**
+ * arm_iommu_attach_device
+ * @dev: valid struct device pointer
+ * @mapping: io address space mapping structure (returned from
+ * arm_iommu_create_mapping)
+ *
+ * Attaches specified io address space mapping to the provided device,
+ * this replaces the dma operations (dma_map_ops pointer) with the
+ * IOMMU aware version. More than one client might be attached to
+ * the same io address space mapping.
+ */
+int arm_iommu_attach_device(struct device *dev,
+ struct dma_iommu_mapping *mapping)
+{
+ int err;
+
+ err = iommu_attach_device(mapping->domain, dev);
+ if (err)
+ return err;
+
+ kref_get(&mapping->kref);
+ dev->archdata.mapping = mapping;
+ set_dma_ops(dev, &iommu_ops);
+
+ pr_info("Attached IOMMU controller to %s device.\n", dev_name(dev));
+ return 0;
+}
+
+#endif
#include <linux/highmem.h>
#include <linux/gfp.h>
#include <linux/memblock.h>
+#include <linux/dma-contiguous.h>
#include <asm/mach-types.h>
#include <asm/memblock.h>
}
#endif
+void __init setup_dma_zone(struct machine_desc *mdesc)
+{
+#ifdef CONFIG_ZONE_DMA
+ if (mdesc->dma_zone_size) {
+ arm_dma_zone_size = mdesc->dma_zone_size;
+ arm_dma_limit = PHYS_OFFSET + arm_dma_zone_size - 1;
+ } else
+ arm_dma_limit = 0xffffffff;
+#endif
+}
+
static void __init arm_bootmem_free(unsigned long min, unsigned long max_low,
unsigned long max_high)
{
* Adjust the sizes according to any special requirements for
* this machine type.
*/
- if (arm_dma_zone_size) {
+ if (arm_dma_zone_size)
arm_adjust_dma_zone(zone_size, zhole_size,
arm_dma_zone_size >> PAGE_SHIFT);
- arm_dma_limit = PHYS_OFFSET + arm_dma_zone_size - 1;
- } else
- arm_dma_limit = 0xffffffff;
#endif
free_area_init_node(0, zone_size, min, zhole_size);
if (mdesc->reserve)
mdesc->reserve();
+ /*
+ * reserve memory for DMA contigouos allocations,
+ * must come from DMA area inside low memory
+ */
+ dma_contiguous_reserve(min(arm_dma_limit, arm_lowmem_limit));
+
arm_memblock_steal_permitted = false;
memblock_allow_resize();
memblock_dump_all();
#define arm_dma_limit ((u32)~0)
#endif
+extern phys_addr_t arm_lowmem_limit;
+
void __init bootmem_init(void);
void arm_mm_memblock_reserve(void);
+void dma_contiguous_remap(void);
PMD_SECT_UNCACHED | PMD_SECT_XN,
.domain = DOMAIN_KERNEL,
},
+ [MT_MEMORY_DMA_READY] = {
+ .prot_pte = L_PTE_PRESENT | L_PTE_YOUNG | L_PTE_DIRTY,
+ .prot_l1 = PMD_TYPE_TABLE,
+ .domain = DOMAIN_KERNEL,
+ },
};
const struct mem_type *get_mem_type(unsigned int type)
if (arch_is_coherent() && cpu_is_xsc3()) {
mem_types[MT_MEMORY].prot_sect |= PMD_SECT_S;
mem_types[MT_MEMORY].prot_pte |= L_PTE_SHARED;
+ mem_types[MT_MEMORY_DMA_READY].prot_pte |= L_PTE_SHARED;
mem_types[MT_MEMORY_NONCACHED].prot_sect |= PMD_SECT_S;
mem_types[MT_MEMORY_NONCACHED].prot_pte |= L_PTE_SHARED;
}
mem_types[MT_DEVICE_CACHED].prot_pte |= L_PTE_SHARED;
mem_types[MT_MEMORY].prot_sect |= PMD_SECT_S;
mem_types[MT_MEMORY].prot_pte |= L_PTE_SHARED;
+ mem_types[MT_MEMORY_DMA_READY].prot_pte |= L_PTE_SHARED;
mem_types[MT_MEMORY_NONCACHED].prot_sect |= PMD_SECT_S;
mem_types[MT_MEMORY_NONCACHED].prot_pte |= L_PTE_SHARED;
}
mem_types[MT_HIGH_VECTORS].prot_l1 |= ecc_mask;
mem_types[MT_MEMORY].prot_sect |= ecc_mask | cp->pmd;
mem_types[MT_MEMORY].prot_pte |= kern_pgprot;
+ mem_types[MT_MEMORY_DMA_READY].prot_pte |= kern_pgprot;
mem_types[MT_MEMORY_NONCACHED].prot_sect |= ecc_mask;
mem_types[MT_ROM].prot_sect |= cp->pmd;
* L1 entries, whereas PGDs refer to a group of L1 entries making
* up one logical pointer to an L2 table.
*/
- if (((addr | end | phys) & ~SECTION_MASK) == 0) {
+ if (type->prot_sect && ((addr | end | phys) & ~SECTION_MASK) == 0) {
pmd_t *p = pmd;
#ifndef CONFIG_ARM_LPAE
}
early_param("vmalloc", early_vmalloc);
-static phys_addr_t lowmem_limit __initdata = 0;
+phys_addr_t arm_lowmem_limit __initdata = 0;
void __init sanity_check_meminfo(void)
{
bank->size = newsize;
}
#endif
- if (!bank->highmem && bank->start + bank->size > lowmem_limit)
- lowmem_limit = bank->start + bank->size;
+ if (!bank->highmem && bank->start + bank->size > arm_lowmem_limit)
+ arm_lowmem_limit = bank->start + bank->size;
j++;
}
}
#endif
meminfo.nr_banks = j;
- high_memory = __va(lowmem_limit - 1) + 1;
- memblock_set_current_limit(lowmem_limit);
+ high_memory = __va(arm_lowmem_limit - 1) + 1;
+ memblock_set_current_limit(arm_lowmem_limit);
}
static inline void prepare_page_table(void)
* Find the end of the first block of lowmem.
*/
end = memblock.memory.regions[0].base + memblock.memory.regions[0].size;
- if (end >= lowmem_limit)
- end = lowmem_limit;
+ if (end >= arm_lowmem_limit)
+ end = arm_lowmem_limit;
/*
* Clear out all the kernel space mappings, except for the first
phys_addr_t end = start + reg->size;
struct map_desc map;
- if (end > lowmem_limit)
- end = lowmem_limit;
+ if (end > arm_lowmem_limit)
+ end = arm_lowmem_limit;
if (start >= end)
break;
{
void *zero_page;
- memblock_set_current_limit(lowmem_limit);
+ memblock_set_current_limit(arm_lowmem_limit);
build_mem_type_table();
prepare_page_table();
map_lowmem();
+ dma_contiguous_remap();
devicemaps_init(mdesc);
kmap_init();
struct list_head vm_list;
unsigned long vm_start;
unsigned long vm_end;
- struct page *vm_pages;
+ void *priv;
int vm_active;
const void *caller;
};
Common code for power management support on S5P and newer SoCs
Note: Do not select this for S5P6440 and S5P6450.
-comment "System MMU"
-
-config S5P_SYSTEM_MMU
- bool "S5P SYSTEM MMU"
- depends on ARCH_EXYNOS4
- help
- Say Y here if you want to enable System MMU
-
config S5P_SLEEP
bool
help
obj-y += irq.o
obj-$(CONFIG_S5P_EXT_INT) += irq-eint.o
obj-$(CONFIG_S5P_GPIO_INT) += irq-gpioint.o
-obj-$(CONFIG_S5P_SYSTEM_MMU) += sysmmu.o
obj-$(CONFIG_S5P_PM) += pm.o irq-pm.o
obj-$(CONFIG_S5P_SLEEP) += sleep.o
obj-$(CONFIG_S5P_HRT) += s5p-time.o
--- /dev/null
+/*
+ * Copyright (C) 2011 Samsung Electronics Co., Ltd.
+ *
+ * Samsung S5P series DP device support
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef PLAT_S5P_DP_H_
+#define PLAT_S5P_DP_H_ __FILE__
+
+#include <video/exynos_dp.h>
+
+extern void s5p_dp_phy_init(void);
+extern void s5p_dp_phy_exit(void);
+
+#endif /* PLAT_S5P_DP_H_ */
--- /dev/null
+/* linux/arch/arm/plat-s5p/include/plat/fimg2d.h
+ *
+ * Copyright (c) 2010 Samsung Electronics Co., Ltd.
+ * http://www.samsung.com/
+ *
+ * Platform Data Structure for Samsung Graphics 2D Hardware
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#ifndef __ASM_ARCH_FIMG2D_H
+#define __ASM_ARCH_FIMG2D_H __FILE__
+
+struct fimg2d_platdata {
+ int hw_ver;
+ const char *parent_clkname;
+ const char *clkname;
+ const char *gate_clkname;
+ unsigned long clkrate;
+};
+
+extern void __init s5p_fimg2d_set_platdata(struct fimg2d_platdata *pd);
+
+#endif /* __ASM_ARCH_FIMG2D_H */
--- /dev/null
+/* linux/arch/arm/plat-s5p/include/plat/tvout.h
+ *
+ * Copyright (c) 2010 Samsung Electronics Co., Ltd.
+ * http://www.samsung.com/
+ *
+ * Platform Header file for Samsung TV driver
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#ifndef __ARM_PLAT_TVOUT_H
+#define __ARM_PLAT_TVOUT_H __FILE__
+
+struct platform_device;
+
+struct s5p_platform_hpd {
+ void (*int_src_hdmi_hpd)(struct platform_device *pdev);
+ void (*int_src_ext_hpd)(struct platform_device *pdev);
+ int (*read_gpio)(struct platform_device *pdev);
+};
+
+extern void s5p_hdmi_hpd_set_platdata(struct s5p_platform_hpd *pd);
+
+/* defined by architecture to configure gpio */
+extern void s5p_int_src_hdmi_hpd(struct platform_device *pdev);
+extern void s5p_int_src_ext_hpd(struct platform_device *pdev);
+extern void s5p_v4l2_int_src_hdmi_hpd(void);
+extern void s5p_v4l2_int_src_ext_hpd(void);
+extern int s5p_hpd_read_gpio(struct platform_device *pdev);
+extern int s5p_v4l2_hpd_read_gpio(void);
+
+struct s5p_platform_cec {
+
+ void (*cfg_gpio)(struct platform_device *pdev);
+};
+
+extern void s5p_hdmi_cec_set_platdata(struct s5p_platform_cec *pd);
+
+/* defined by architecture to configure gpio */
+extern void s5p_cec_cfg_gpio(struct platform_device *pdev);
+
+extern void s5p_tv_setup(void);
+
+#endif /* __ASM_PLAT_TV_HPD_H */
default "0" if DEBUG_S3C_UART0
default "1" if DEBUG_S3C_UART1
default "2" if DEBUG_S3C_UART2
+ default "3" if DEBUG_S3C_UART3
endif
#include <plat/regs-serial.h>
#include <plat/regs-spi.h>
#include <plat/s3c64xx-spi.h>
+#include <plat/fimg2d.h>
static u64 samsung_device_dma_mask = DMA_BIT_MASK(32);
};
#endif /* CONFIG_S5P_DEV_FIMC3 */
-/* G2D */
-
-#ifdef CONFIG_S5P_DEV_G2D
-static struct resource s5p_g2d_resource[] = {
- [0] = {
- .start = S5P_PA_G2D,
- .end = S5P_PA_G2D + SZ_4K - 1,
- .flags = IORESOURCE_MEM,
- },
- [1] = {
- .start = IRQ_2D,
- .end = IRQ_2D,
- .flags = IORESOURCE_IRQ,
- },
-};
-
-struct platform_device s5p_device_g2d = {
- .name = "s5p-g2d",
- .id = 0,
- .num_resources = ARRAY_SIZE(s5p_g2d_resource),
- .resource = s5p_g2d_resource,
- .dev = {
- .dma_mask = &samsung_device_dma_mask,
- .coherent_dma_mask = DMA_BIT_MASK(32),
- },
-};
-#endif /* CONFIG_S5P_DEV_G2D */
-
#ifdef CONFIG_S5P_DEV_JPEG
static struct resource s5p_jpeg_resource[] = {
[0] = DEFINE_RES_MEM(S5P_PA_JPEG, SZ_4K),
}
#endif /* CONFIG_S5P_DEV_FIMD0 */
+/* G2D */
+
+#ifdef CONFIG_S5P_DEV_G2D
+static struct resource s5p_g2d_resource[] = {
+ [0] = DEFINE_RES_MEM(S5P_PA_G2D, SZ_4K),
+ [1] = DEFINE_RES_IRQ(IRQ_2D),
+};
+
+struct platform_device s5p_device_g2d = {
+ .name = "s5p-g2d",
+ .id = 0,
+ .num_resources = ARRAY_SIZE(s5p_g2d_resource),
+ .resource = s5p_g2d_resource,
+ .dev = {
+ .dma_mask = &samsung_device_dma_mask,
+ .coherent_dma_mask = DMA_BIT_MASK(32),
+ },
+};
+
+#ifdef CONFIG_VIDEO_FIMG2D4X
+static struct fimg2d_platdata default_g2d_data __initdata = {
+ .parent_clkname = "mout_g2d0",
+ .clkname = "sclk_fimg2d",
+ .gate_clkname = "fimg2d",
+ .clkrate = 250 * 1000000,
+};
+
+void __init s5p_fimg2d_set_platdata(struct fimg2d_platdata *pd)
+{
+ struct fimg2d_platdata *npd;
+
+ if (!pd)
+ pd = &default_g2d_data;
+
+ npd = kmemdup(pd, sizeof(*pd), GFP_KERNEL);
+ if (!npd)
+ printk(KERN_ERR "no memory for fimg2d platform data\n");
+ else
+ s5p_device_g2d.dev.platform_data = npd;
+}
+#endif /* CONFIG_VIDEO_FIMG2D4X */
+#endif /* CONFIG_S5P_DEV_G2D */
+
/* HWMON */
#ifdef CONFIG_S3C_DEV_HWMON
};
struct platform_device s3c64xx_device_spi0 = {
- .name = "s3c64xx-spi",
+ .name = "s3c6410-spi",
.id = 0,
.num_resources = ARRAY_SIZE(s3c64xx_spi0_resource),
.resource = s3c64xx_spi0_resource,
},
};
-void __init s3c64xx_spi0_set_platdata(struct s3c64xx_spi_info *pd,
- int src_clk_nr, int num_cs)
+void __init s3c64xx_spi0_set_platdata(int (*cfg_gpio)(void), int src_clk_nr,
+ int num_cs)
{
- if (!pd) {
- pr_err("%s:Need to pass platform data\n", __func__);
- return;
- }
+ struct s3c64xx_spi_info pd;
/* Reject invalid configuration */
if (!num_cs || src_clk_nr < 0) {
return;
}
- pd->num_cs = num_cs;
- pd->src_clk_nr = src_clk_nr;
- if (!pd->cfg_gpio)
- pd->cfg_gpio = s3c64xx_spi0_cfg_gpio;
+ pd.num_cs = num_cs;
+ pd.src_clk_nr = src_clk_nr;
+ pd.cfg_gpio = (cfg_gpio) ? cfg_gpio : s3c64xx_spi0_cfg_gpio;
- s3c_set_platdata(pd, sizeof(*pd), &s3c64xx_device_spi0);
+ s3c_set_platdata(&pd, sizeof(pd), &s3c64xx_device_spi0);
}
#endif /* CONFIG_S3C64XX_DEV_SPI0 */
};
struct platform_device s3c64xx_device_spi1 = {
- .name = "s3c64xx-spi",
+ .name = "s3c6410-spi",
.id = 1,
.num_resources = ARRAY_SIZE(s3c64xx_spi1_resource),
.resource = s3c64xx_spi1_resource,
},
};
-void __init s3c64xx_spi1_set_platdata(struct s3c64xx_spi_info *pd,
- int src_clk_nr, int num_cs)
+void __init s3c64xx_spi1_set_platdata(int (*cfg_gpio)(void), int src_clk_nr,
+ int num_cs)
{
- if (!pd) {
- pr_err("%s:Need to pass platform data\n", __func__);
- return;
- }
-
/* Reject invalid configuration */
if (!num_cs || src_clk_nr < 0) {
pr_err("%s: Invalid SPI configuration\n", __func__);
return;
}
- pd->num_cs = num_cs;
- pd->src_clk_nr = src_clk_nr;
- if (!pd->cfg_gpio)
- pd->cfg_gpio = s3c64xx_spi1_cfg_gpio;
+ pd.num_cs = num_cs;
+ pd.src_clk_nr = src_clk_nr;
+ pd.cfg_gpio = (cfg_gpio) ? cfg_gpio : s3c64xx_spi1_cfg_gpio;
- s3c_set_platdata(pd, sizeof(*pd), &s3c64xx_device_spi1);
+ s3c_set_platdata(&pd, sizeof(pd), &s3c64xx_device_spi1);
}
#endif /* CONFIG_S3C64XX_DEV_SPI1 */
};
struct platform_device s3c64xx_device_spi2 = {
- .name = "s3c64xx-spi",
+ .name = "s3c6410-spi",
.id = 2,
.num_resources = ARRAY_SIZE(s3c64xx_spi2_resource),
.resource = s3c64xx_spi2_resource,
},
};
-void __init s3c64xx_spi2_set_platdata(struct s3c64xx_spi_info *pd,
- int src_clk_nr, int num_cs)
+void __init s3c64xx_spi2_set_platdata(int (*cfg_gpio)(void), int src_clk_nr,
+ int num_cs)
{
- if (!pd) {
- pr_err("%s:Need to pass platform data\n", __func__);
- return;
- }
+ struct s3c64xx_spi_info pd;
/* Reject invalid configuration */
if (!num_cs || src_clk_nr < 0) {
return;
}
- pd->num_cs = num_cs;
- pd->src_clk_nr = src_clk_nr;
- if (!pd->cfg_gpio)
- pd->cfg_gpio = s3c64xx_spi2_cfg_gpio;
+ pd.num_cs = num_cs;
+ pd.src_clk_nr = src_clk_nr;
+ pd.cfg_gpio = (cfg_gpio) ? cfg_gpio : s3c64xx_spi2_cfg_gpio;
- s3c_set_platdata(pd, sizeof(*pd), &s3c64xx_device_spi2);
+ s3c_set_platdata(&pd, sizeof(pd), &s3c64xx_device_spi2);
}
#endif /* CONFIG_S3C64XX_DEV_SPI2 */
extern struct platform_device exynos4_device_pcm2;
extern struct platform_device exynos4_device_pd[];
extern struct platform_device exynos4_device_spdif;
-extern struct platform_device exynos4_device_sysmmu;
extern struct platform_device samsung_asoc_dma;
extern struct platform_device samsung_asoc_idma;
--- /dev/null
+/* linux/arm/arch/plat-s5p/include/plat/dsim.h
+ *
+ * Platform data header for Samsung SoC MIPI-DSIM.
+ *
+ * Copyright (c) 2011 Samsung Electronics
+ *
+ * InKi Dae <inki.dae@samsung.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#ifndef _DSIM_H
+#define _DSIM_H
+
+#include <linux/device.h>
+#include <linux/fb.h>
+#include <linux/notifier.h>
+
+#include <linux/regulator/consumer.h>
+
+#define to_dsim_plat(d) (to_platform_device(d)->dev.platform_data)
+
+enum mipi_dsim_interface_type {
+ DSIM_COMMAND,
+ DSIM_VIDEO
+};
+
+enum mipi_dsim_virtual_ch_no {
+ DSIM_VIRTUAL_CH_0,
+ DSIM_VIRTUAL_CH_1,
+ DSIM_VIRTUAL_CH_2,
+ DSIM_VIRTUAL_CH_3
+};
+
+enum mipi_dsim_burst_mode_type {
+ DSIM_NON_BURST_SYNC_EVENT,
+ DSIM_NON_BURST_SYNC_PULSE = 2,
+ DSIM_BURST = 1,
+ DSIM_NON_VIDEO_MODE = 4
+};
+
+enum mipi_dsim_no_of_data_lane {
+ DSIM_DATA_LANE_1,
+ DSIM_DATA_LANE_2,
+ DSIM_DATA_LANE_3,
+ DSIM_DATA_LANE_4
+};
+
+enum mipi_dsim_byte_clk_src {
+ DSIM_PLL_OUT_DIV8,
+ DSIM_EXT_CLK_DIV8,
+ DSIM_EXT_CLK_BYPASS
+};
+
+enum mipi_dsim_pixel_format {
+ DSIM_CMD_3BPP,
+ DSIM_CMD_8BPP,
+ DSIM_CMD_12BPP,
+ DSIM_CMD_16BPP,
+ DSIM_VID_16BPP_565,
+ DSIM_VID_18BPP_666PACKED,
+ DSIM_18BPP_666LOOSELYPACKED,
+ DSIM_24BPP_888
+};
+
+/**
+ * struct mipi_dsim_config - interface for configuring mipi-dsi controller.
+ *
+ * @auto_flush: enable or disable Auto flush of MD FIFO using VSYNC pulse.
+ * @eot_disable: enable or disable EoT packet in HS mode.
+ * @auto_vertical_cnt: specifies auto vertical count mode.
+ * in Video mode, the vertical line transition uses line counter
+ * configured by VSA, VBP, and Vertical resolution.
+ * If this bit is set to '1', the line counter does not use VSA and VBP
+ * registers.(in command mode, this variable is ignored)
+ * @hse: set horizontal sync event mode.
+ * In VSYNC pulse and Vporch area, MIPI DSI master transfers only HSYNC
+ * start packet to MIPI DSI slave at MIPI DSI spec1.1r02.
+ * this bit transfers HSYNC end packet in VSYNC pulse and Vporch area
+ * (in mommand mode, this variable is ignored)
+ * @hfp: specifies HFP disable mode.
+ * if this variable is set, DSI master ignores HFP area in VIDEO mode.
+ * (in command mode, this variable is ignored)
+ * @hbp: specifies HBP disable mode.
+ * if this variable is set, DSI master ignores HBP area in VIDEO mode.
+ * (in command mode, this variable is ignored)
+ * @hsa: specifies HSA disable mode.
+ * if this variable is set, DSI master ignores HSA area in VIDEO mode.
+ * (in command mode, this variable is ignored)
+ * @e_interface: specifies interface to be used.(CPU or RGB interface)
+ * @e_virtual_ch: specifies virtual channel number that main or
+ * sub diaplsy uses.
+ * @e_pixel_format: specifies pixel stream format for main or sub display.
+ * @e_burst_mode: selects Burst mode in Video mode.
+ * in Non-burst mode, RGB data area is filled with RGB data and NULL
+ * packets, according to input bandwidth of RGB interface.
+ * In Burst mode, RGB data area is filled with RGB data only.
+ * @e_no_data_lane: specifies data lane count to be used by Master.
+ * @e_byte_clk: select byte clock source. (it must be DSIM_PLL_OUT_DIV8)
+ * DSIM_EXT_CLK_DIV8 and DSIM_EXT_CLK_BYPASSS are not supported.
+ * @pll_stable_time: specifies the PLL Timer for stability of the ganerated
+ * clock(System clock cycle base)
+ * if the timer value goes to 0x00000000, the clock stable bit of
+status
+ * and interrupt register is set.
+ * @esc_clk: specifies escape clock frequency for getting the escape clock
+ * prescaler value.
+ * @stop_holding_cnt: specifies the interval value between transmitting
+ * read packet(or write "set_tear_on" command) and BTA request.
+ * after transmitting read packet or write "set_tear_on" command,
+ * BTA requests to D-PHY automatically. this counter value specifies
+ * the interval between them.
+ * @bta_timeout: specifies the timer for BTA.
+ * this register specifies time out from BTA request to change
+ * the direction with respect to Tx escape clock.
+ * @rx_timeout: specifies the timer for LP Rx mode timeout.
+ * this register specifies time out on how long RxValid deasserts,
+ * after RxLpdt asserts with respect to Tx escape clock.
+ * - RxValid specifies Rx data valid indicator.
+ * - RxLpdt specifies an indicator that D-PHY is under RxLpdt mode.
+ * - RxValid and RxLpdt specifies signal from D-PHY.
+ * @lcd_panel_info: pointer for lcd panel specific structure.
+ * this structure specifies width, height, timing and polarity and so
+on.
+ * @mipi_ddi_pd: pointer to lcd panel platform data.
+ */
+struct mipi_dsim_config {
+ unsigned char auto_flush;
+ unsigned char eot_disable;
+
+ unsigned char auto_vertical_cnt;
+ unsigned char hse;
+ unsigned char hfp;
+ unsigned char hbp;
+ unsigned char hsa;
+
+ enum mipi_dsim_interface_type e_interface;
+ enum mipi_dsim_virtual_ch_no e_virtual_ch;
+ enum mipi_dsim_pixel_format e_pixel_format;
+ enum mipi_dsim_burst_mode_type e_burst_mode;
+ enum mipi_dsim_no_of_data_lane e_no_data_lane;
+ enum mipi_dsim_byte_clk_src e_byte_clk;
+
+ unsigned char p;
+ unsigned short m;
+ unsigned char s;
+
+ unsigned int pll_stable_time;
+ unsigned long esc_clk;
+
+ unsigned short stop_holding_cnt;
+ unsigned char bta_timeout;
+ unsigned short rx_timeout;
+
+ void *lcd_panel_info;
+ void *dsim_ddi_pd;
+};
+
+/* for RGB Interface */
+struct mipi_dsi_lcd_timing {
+ int left_margin;
+ int right_margin;
+ int upper_margin;
+ int lower_margin;
+ int hsync_len;
+ int vsync_len;
+};
+
+/* for CPU Interface */
+struct mipi_dsi_cpu_timing {
+ unsigned int cs_setup;
+ unsigned int wr_setup;
+ unsigned int wr_act;
+ unsigned int wr_hold;
+};
+
+struct mipi_dsi_lcd_size {
+ unsigned int width;
+ unsigned int height;
+};
+
+struct mipi_dsim_lcd_config {
+ enum mipi_dsim_interface_type e_interface;
+ unsigned int parameter[3];
+
+ /* lcd panel info */
+ struct mipi_dsi_lcd_timing rgb_timing;
+ struct mipi_dsi_cpu_timing cpu_timing;
+ struct mipi_dsi_lcd_size lcd_size;
+ /* platform data for lcd panel based on MIPI-DSI. */
+ void *mipi_ddi_pd;
+};
+
+/**
+ * struct mipi_dsim_device - global interface for mipi-dsi driver.
+ *
+ * @dev: driver model representation of the device.
+ * @clock: pointer to MIPI-DSI clock of clock framework.
+ * @irq: interrupt number to MIPI-DSI controller.
+ * @reg_base: base address to memory mapped SRF of MIPI-DSI controller.
+ * (virtual address)
+ * @pd: pointer to MIPI-DSI driver platform data.
+ * @dsim_info: infomation for configuring mipi-dsi controller.
+ * @master_ops: callbacks to mipi-dsi operations.
+ * @lcd_info: pointer to mipi_lcd_info structure.
+ * @state: specifies status of MIPI-DSI controller.
+ * the status could be RESET, INIT, STOP, HSCLKEN and ULPS.
+ * @data_lane: specifiec enabled data lane number.
+ * this variable would be set by driver according to e_no_data_lane
+ * automatically.
+ * @e_clk_src: select byte clock source.
+ * this variable would be set by driver according to e_byte_clock
+ * automatically.
+ * @hs_clk: HS clock rate.
+ * this variable would be set by driver automatically.
+ * @byte_clk: Byte clock rate.
+ * this variable would be set by driver automatically.
+ * @escape_clk: ESCAPE clock rate.
+ * this variable would be set by driver automatically.
+ * @freq_band: indicates Bitclk frequency band for D-PHY global timing.
+ * Serial Clock(=ByteClk X 8) FreqBand[3:0]
+ * ~ 99.99 MHz 0000
+ * 100 ~ 119.99 MHz 0001
+ * 120 ~ 159.99 MHz 0010
+ * 160 ~ 199.99 MHz 0011
+ * 200 ~ 239.99 MHz 0100
+ * 140 ~ 319.99 MHz 0101
+ * 320 ~ 389.99 MHz 0110
+ * 390 ~ 449.99 MHz 0111
+ * 450 ~ 509.99 MHz 1000
+ * 510 ~ 559.99 MHz 1001
+ * 560 ~ 639.99 MHz 1010
+ * 640 ~ 689.99 MHz 1011
+ * 690 ~ 769.99 MHz 1100
+ * 770 ~ 869.99 MHz 1101
+ * 870 ~ 949.99 MHz 1110
+ * 950 ~ 1000 MHz 1111
+ * this variable would be calculated by driver automatically.
+ */
+struct mipi_dsim_device {
+ struct device *dev;
+ struct resource *res;
+ struct clk *clock;
+ unsigned int irq;
+ void __iomem *reg_base;
+
+ struct s5p_platform_mipi_dsim *pd;
+ struct mipi_dsim_config *dsim_config;
+
+ unsigned int state;
+ unsigned int data_lane;
+ enum mipi_dsim_byte_clk_src e_clk_src;
+ unsigned long hs_clk;
+ unsigned long byte_clk;
+ unsigned long escape_clk;
+ unsigned char freq_band;
+ unsigned char id;
+ struct notifier_block fb_notif;
+
+ struct mipi_dsim_lcd_driver *dsim_lcd_drv;
+};
+
+/**
+ * struct s5p_platform_mipi_dsim - interface to platform data
+ * for mipi-dsi driver.
+ *
+ * @mipi_dsim_config: pointer of structure for configuring mipi-dsi controller.
+ * @dsim_lcd_info: pointer to structure for configuring
+ * mipi-dsi based lcd panel.
+ * @mipi_power: callback pointer for enabling or disabling mipi power.
+ * @part_reset: callback pointer for reseting mipi phy.
+ * @init_d_phy: callback pointer for enabing d_phy of dsi master.
+ * @get_fb_frame_done: callback pointer for getting frame done status of
+the
+ * display controller(FIMD).
+ * @trigger: callback pointer for triggering display controller(FIMD)
+ * in case of CPU mode.
+ * @delay_for_stabilization: specifies stable time.
+ * this delay needs when writing data on SFR
+ * after mipi mode became LP mode.
+ */
+struct s5p_platform_mipi_dsim {
+ const char clk_name[16];
+
+ struct mipi_dsim_config *dsim_config;
+ struct mipi_dsim_lcd_config *dsim_lcd_config;
+
+ unsigned int delay_for_stabilization;
+
+ int (*mipi_power) (struct mipi_dsim_device *dsim, unsigned int
+ enable);
+ int (*part_reset) (struct mipi_dsim_device *dsim);
+ int (*init_d_phy) (struct mipi_dsim_device *dsim, unsigned int enable);
+ int (*get_fb_frame_done) (struct fb_info *info);
+ void (*trigger) (struct fb_info *info);
+};
+
+/**
+ * driver structure for mipi-dsi based lcd panel.
+ *
+ * this structure should be registered by lcd panel driver.
+ * mipi-dsi driver seeks lcd panel registered through name field
+ * and calls these callback functions in appropriate time.
+ */
+
+struct mipi_dsim_lcd_driver {
+ int (*probe)(struct mipi_dsim_device *dsim);
+ int (*suspend)(struct mipi_dsim_device *dsim);
+ int (*displayon)(struct mipi_dsim_device *dsim);
+ int (*resume)(struct mipi_dsim_device *dsim);
+};
+
+/**
+ * register mipi_dsim_lcd_driver object defined by lcd panel driver
+ * to mipi-dsi driver.
+ */
+extern int s5p_dsim_part_reset(struct mipi_dsim_device *dsim);
+extern int s5p_dsim_init_d_phy(struct mipi_dsim_device *dsim,
+ unsigned int enable);
+
+#endif /* _DSIM_H */
*/
#define S3C_FB_MAX_WIN (5)
+/* IOCTL commands */
+#define S3CFB_WIN_POSITION _IOW('F', 203, \
+ struct s3c_fb_user_window)
+#define S3CFB_WIN_SET_PLANE_ALPHA _IOW('F', 204, \
+ struct s3c_fb_user_plane_alpha)
+#define S3CFB_WIN_SET_CHROMA _IOW('F', 205, \
+ struct s3c_fb_user_chroma)
+#define S3CFB_SET_VSYNC_INT _IOW('F', 206, u32)
+
+#define S3CFB_GET_ION_USER_HANDLE _IOWR('F', 208, \
+ struct s3c_fb_user_ion_client)
+
/**
* struct s3c_fb_pd_win - per window setup data
* @win_mode: The display parameters to initialise (not for window 0)
unsigned short max_bpp;
unsigned short virtual_x;
unsigned short virtual_y;
+ unsigned short width;
+ unsigned short height;
};
/**
* the data from the display system to the connected display
* device.
* @default_win: default window layer number to be used for UI layer.
+ * @clock_rate: To Setup FIMD source clock rate.
* @vidcon0: The base vidcon0 values to control the panel data format.
* @vidcon1: The base vidcon1 values to control the panel data output.
* @win: The setup data for each hardware window, or NULL for unused.
struct s3c_fb_pd_win *win[S3C_FB_MAX_WIN];
u32 default_win;
-
+ u32 clock_rate;
u32 vidcon0;
u32 vidcon1;
};
*/
extern void s5p64x0_fb_gpio_setup_24bpp(void);
+/**
+ * exynos4_fimd0_setup_clock() = Exynos4 setup function for parent clock.
+ * @dev: device pointer
+ * @parent: parent clock used for LCD pixel clock
+ * @clk_rate: clock rate for parent clock
+ */
+int __init exynos4_fimd0_setup_clock(struct device *dev, const char *parent,
+ unsigned long clk_rate);
+
+int __init exynos4_fimd_setup_clock(struct device *dev, const char *bus_clk,
+ const char *parent, unsigned long clk_rate);
#endif /* __PLAT_S3C_FB_H */
#define S3C_VA_USB_HSPHY S3C64XX_VA_USB_HSPHY
+#define S5P_VA_DRD_PHY S3C_ADDR_CPU(0x00300000)
/*
* ISA style IO, for each machine to sort out mappings for,
* if it implements it. We reserve two 16M regions for ISA.
#define S5P_VA_GIC_CPU S3C_ADDR(0x02810000)
#define S5P_VA_GIC_DIST S3C_ADDR(0x02820000)
+#define S5P_VA_AUDSS S3C_ADDR(0x02910000)
+
#define VA_VIC(x) (S3C_VA_IRQ + ((x) * 0x10000))
#define VA_VIC0 VA_VIC(0)
#define VA_VIC1 VA_VIC(1)
--- /dev/null
+/* linux/arm/arch/mach-s5pc110/include/plat/mipi_dsi.h
+ *
+ *
+ * Copyright (c) 2011 Samsung Electronics
+ * InKi Dae <inki.dae <at> samsung.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#ifndef _MIPI_DSI_H
+#define _MIPI_DSI_H
+
+#if defined(CONFIG_LCD_MIPI_TC358764)
+extern struct mipi_dsim_lcd_driver tc358764_mipi_lcd_driver;
+#endif
+
+extern int s5p_mipi_dsi_wr_data(struct mipi_dsim_device *dsim,
+ unsigned int data_id, unsigned int data0, unsigned int data1);
+
+enum mipi_ddi_interface {
+ RGB_IF = 0x4000,
+ I80_IF = 0x8000,
+ YUV_601 = 0x10000,
+ YUV_656 = 0x20000,
+ MIPI_VIDEO = 0x1000,
+ MIPI_COMMAND = 0x2000,
+};
+
+enum mipi_ddi_panel_select {
+ DDI_MAIN_LCD = 0,
+ DDI_SUB_LCD = 1,
+};
+
+enum mipi_ddi_model {
+ S6DR117 = 0,
+};
+
+enum mipi_ddi_parameter {
+ /* DSIM video interface parameter */
+ DSI_VIRTUAL_CH_ID = 0,
+ DSI_FORMAT = 1,
+ DSI_VIDEO_MODE_SEL = 2,
+};
+
+#endif /* _MIPI_DSI_H */
#define VIDCON1_FSTATUS_EVEN (1 << 15)
/* Video timing controls */
+#ifdef CONFIG_FB_EXYNOS_FIMD_V8
+#define VIDTCON0 (0x20010)
+#define VIDTCON1 (0x20014)
+#define VIDTCON3 (0x2001C)
+#else
#define VIDTCON0 (0x10)
#define VIDTCON1 (0x14)
-#define VIDTCON2 (0x18)
+#define VIDTCON3 (0x1C)
+#endif
/* Window position controls */
#define VIDOSD_BASE (0x40)
#define VIDINTCON0 (0x130)
+#define VIDINTCON1 (0x134)
/* WINCONx */
+#define WINCONx_CSC_CON_EQ709 (1 << 28)
+#define WINCONx_CSC_CON_EQ601 (0 << 28)
#define WINCONx_CSCWIDTH_MASK (0x3 << 26)
#define WINCONx_CSCWIDTH_SHIFT (26)
#define WINCONx_CSCWIDTH_WIDE (0x0 << 26)
#define VIDCON0 (0x00)
#define VIDCON0_INTERLACE (1 << 29)
+
+#ifdef CONFIG_FB_EXYNOS_FIMD_V8
+#define VIDOUT_CON (0x20000)
+#define VIDOUT_CON_VIDOUT_UP_MASK (0x1 << 16)
+#define VIDOUT_CON_VIDOUT_UP_SHIFT (16)
+#define VIDOUT_CON_VIDOUT_UP_ALWAYS (0x0 << 16)
+#define VIDOUT_CON_VIDOUT_UP_START_FRAME (0x1 << 16)
+#define VIDOUT_CON_VIDOUT_F_MASK (0x7 << 8)
+#define VIDOUT_CON_VIDOUT_F_SHIFT (8)
+#define VIDOUT_CON_VIDOUT_F_RGB (0x0 << 8)
+#define VIDOUT_CON_VIDOUT_F_I80_LDI0 (0x2 << 8)
+#define VIDOUT_CON_VIDOUT_F_I80_LDI1 (0x3 << 8)
+#define VIDOUT_CON_VIDOUT_F_WB (0x4 << 8)
+#endif
+
#define VIDCON0_VIDOUT_MASK (0x3 << 26)
#define VIDCON0_VIDOUT_SHIFT (26)
#define VIDCON0_VIDOUT_RGB (0x0 << 26)
#define VIDCON0_VIDOUT_TV (0x1 << 26)
#define VIDCON0_VIDOUT_I80_LDI0 (0x2 << 26)
#define VIDCON0_VIDOUT_I80_LDI1 (0x3 << 26)
+#define VIDCON0_VIDOUT_WB (0x4 << 26)
#define VIDCON0_L1_DATA_MASK (0x7 << 23)
#define VIDCON0_L1_DATA_SHIFT (23)
#define VIDCON0_ENVID (1 << 1)
#define VIDCON0_ENVID_F (1 << 0)
+#ifdef CONFIG_FB_EXYNOS_FIMD_V8
+#define VIDOUT_CON (0x20000)
+#define VIDCON1 (0x20004)
+#else
#define VIDCON1 (0x04)
+#endif
+
#define VIDCON1_LINECNT_MASK (0x7ff << 16)
#define VIDCON1_LINECNT_SHIFT (16)
#define VIDCON1_LINECNT_GET(_v) (((_v) >> 16) & 0x7ff)
#define VIDCON2_TVFMTSEL1_RGB (0x0 << 12)
#define VIDCON2_TVFMTSEL1_YUV422 (0x1 << 12)
#define VIDCON2_TVFMTSEL1_YUV444 (0x2 << 12)
+#define VIDCON2_TVFMTSEL1_SHIFT (12)
+#define VIDCON2_TVFMTSEL_SW (1 << 14)
+#define VIDCON2_TVFORMATSEL_YUV444 (0x2 << 12)
+
+#define VIDCON2_TVFMTSEL1_MASK (0x3 << 12)
+#define VIDCON2_TVFMTSEL1_RGB (0x0 << 12)
+#define VIDCON2_TVFMTSEL1_YUV422 (0x1 << 12)
+#define VIDCON2_TVFMTSEL1_YUV444 (0x2 << 12)
#define VIDCON2_ORGYCbCr (1 << 8)
#define VIDCON2_YUVORDCrCb (1 << 7)
#define VIDTCON1_HSPW_SHIFT (0)
#define VIDTCON1_HSPW_LIMIT (0xff)
#define VIDTCON1_HSPW(_x) ((_x) << 0)
+#define VIDCON1_VCLK_MASK (0x3 << 9)
+#define VIDCON1_VCLK_HOLD (0x0 << 9)
+#define VIDCON1_VCLK_RUN (0x1 << 9)
+#ifdef CONFIG_FB_EXYNOS_FIMD_V8
+#define VIDTCON2 (0x20018)
+#else
#define VIDTCON2 (0x18)
+#endif
#define VIDTCON2_LINEVAL_E(_x) ((((_x) & 0x800) >> 11) << 23)
#define VIDTCON2_LINEVAL_MASK (0x7ff << 11)
#define VIDTCON2_LINEVAL_SHIFT (11)
#define WINCONx_BYTSWP (1 << 17)
#define WINCONx_HAWSWP (1 << 16)
#define WINCONx_WSWP (1 << 15)
+#define WINCONx_ENLOCAL_MASK (0xf << 15)
+#define WINCONx_INRGB_RGB (0 << 13)
+#define WINCONx_INRGB_YCBCR (1 << 13)
#define WINCONx_BURSTLEN_MASK (0x3 << 9)
#define WINCONx_BURSTLEN_SHIFT (9)
#define WINCONx_BURSTLEN_16WORD (0x0 << 9)
#define WINCON0_BPPMODE_24BPP_888 (0xb << 2)
#define WINCON1_BLD_PIX (1 << 6)
+#define WINCON1_BLD_PLANE (0 << 6)
#define WINCON1_ALPHA_SEL (1 << 1)
#define WINCON1_BPPMODE_MASK (0xf << 2)
#define WPALCON_W0PAL_16BPP_A555 (0x5 << 0)
#define WPALCON_W0PAL_16BPP_565 (0x6 << 0)
+/* Clock gate mode control */
+#define REG_CLKGATE_MODE (0x1b0)
+#define REG_CLKGATE_MODE_AUTO_CLOCK_GATE (0 << 0)
+#define REG_CLKGATE_MODE_NON_CLOCK_GATE (1 << 0)
+
/* Blending equation control */
#define BLENDCON (0x260)
#define BLENDCON_NEW_MASK (1 << 0)
#define BLENDCON_NEW_8BIT_ALPHA_VALUE (1 << 0)
#define BLENDCON_NEW_4BIT_ALPHA_VALUE (0 << 0)
+/* Window alpha control */
+#define VIDW0ALPHA0 (0x200)
+#define VIDW0ALPHA1 (0x204)
+#define DPCLKCON (0x27c)
+#define DPCLKCON_ENABLE (1 << 1)
--- /dev/null
+/* linux/arch/arm/plat-s5p/include/plat/regs-mipidsim.h
+ *
+ * Register definition file for Samsung MIPI-DSIM driver
+ *
+ * Copyright (c) 2011 Samsung Electronics
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#ifndef _REGS_MIPIDSIM_H
+#define _REGS_MIPIDSIM_H
+
+#define S5P_DSIM_STATUS (0x0) /* Status register */
+#define S5P_DSIM_SWRST (0x4) /* Software reset register */
+#define S5P_DSIM_CLKCTRL (0x8) /* Clock control register */
+#define S5P_DSIM_TIMEOUT (0xc) /* Time out register */
+#define S5P_DSIM_CONFIG (0x10) /* Configuration register */
+#define S5P_DSIM_ESCMODE (0x14) /* Escape mode register */
+
+/* Main display image resolution register */
+#define S5P_DSIM_MDRESOL (0x18)
+#define S5P_DSIM_MVPORCH (0x1c) /* Main display Vporch register */
+#define S5P_DSIM_MHPORCH (0x20) /* Main display Hporch register */
+#define S5P_DSIM_MSYNC (0x24) /* Main display sync area register
+*/
+
+/* Sub display image resolution register */
+#define S5P_DSIM_SDRESOL (0x28)
+#define S5P_DSIM_INTSRC (0x2c) /* Interrupt source
+register */
+#define S5P_DSIM_INTMSK (0x30) /* Interrupt mask register
+*/
+#define S5P_DSIM_PKTHDR (0x34) /* Packet Header FIFO
+register */
+#define S5P_DSIM_PAYLOAD (0x38) /* Payload FIFO register */
+#define S5P_DSIM_RXFIFO (0x3c) /* Read FIFO register */
+#define S5P_DSIM_FIFOTHLD (0x40) /* FIFO threshold level register */
+#define S5P_DSIM_FIFOCTRL (0x44) /* FIFO status and control register
+*/
+
+/* FIFO memory AC characteristic register */
+#define S5P_DSIM_PLLCTRL (0x4c) /* PLL control register */
+#define S5P_DSIM_PLLTMR (0x50) /* PLL timer register */
+#define S5P_DSIM_PHYACCHR (0x54) /* D-PHY AC characteristic register
+*/
+#define S5P_DSIM_PHYACCHR1 (0x58) /* D-PHY AC characteristic
+register1 */
+
+/* DSIM_STATUS */
+#define DSIM_STOP_STATE_DAT(x) (((x) & 0xf) << 0)
+#define DSIM_STOP_STATE_CLK (1 << 8)
+#define DSIM_TX_READY_HS_CLK (1 << 10)
+
+/* DSIM_SWRST */
+#define DSIM_FUNCRST (1 << 16)
+#define DSIM_SWRST (1 << 0)
+
+/* S5P_DSIM_TIMEOUT */
+#define DSIM_LPDR_TOUT_SHIFT (0)
+#define DSIM_BTA_TOUT_SHIFT (16)
+
+/* S5P_DSIM_CLKCTRL */
+#define DSIM_LANE_ESC_CLKEN_SHIFT (19)
+#define DSIM_BYTE_CLKEN_SHIFT (24)
+#define DSIM_BYTE_CLK_SRC_SHIFT (25)
+#define DSIM_PLL_BYPASS_SHIFT (27)
+#define DSIM_ESC_CLKEN_SHIFT (28)
+#define DSIM_TX_REQUEST_HSCLK_SHIFT (31)
+#define DSIM_LANE_ESC_CLKEN(x) (((x) & 0x1f) << \
+ DSIM_LANE_ESC_CLKEN_SHIFT)
+#define DSIM_BYTE_CLK_ENABLE (1 << DSIM_BYTE_CLKEN_SHIFT)
+#define DSIM_BYTE_CLK_DISABLE (0 << DSIM_BYTE_CLKEN_SHIFT)
+#define DSIM_PLL_BYPASS_EXTERNAL (1 << DSIM_PLL_BYPASS_SHIFT)
+#define DSIM_ESC_CLKEN_ENABLE (1 << DSIM_ESC_CLKEN_SHIFT)
+#define DSIM_ESC_CLKEN_DISABLE (0 << DSIM_ESC_CLKEN_SHIFT)
+
+/* S5P_DSIM_CONFIG */
+#define DSIM_NUM_OF_DATALANE_SHIFT (5)
+#define DSIM_HSA_MODE_SHIFT (20)
+#define DSIM_HBP_MODE_SHIFT (21)
+#define DSIM_HFP_MODE_SHIFT (22)
+#define DSIM_HSE_MODE_SHIFT (23)
+#define DSIM_AUTO_MODE_SHIFT (24)
+#define DSIM_LANE_ENx(x) (((x) & 0x1f) << 0)
+
+#define DSIM_NUM_OF_DATA_LANE(x) ((x) << DSIM_NUM_OF_DATALANE_SHIFT)
+
+/* S5P_DSIM_ESCMODE */
+#define DSIM_TX_LPDT_SHIFT (6)
+#define DSIM_CMD_LPDT_SHIFT (7)
+#define DSIM_TX_LPDT_LP (1 << DSIM_TX_LPDT_SHIFT)
+#define DSIM_CMD_LPDT_LP (1 << DSIM_CMD_LPDT_SHIFT)
+#define DSIM_STOP_STATE_CNT_SHIFT (21)
+#define DSIM_FORCE_STOP_STATE_SHIFT (20)
+
+/* S5P_DSIM_MDRESOL */
+#define DSIM_MAIN_STAND_BY (1 << 31)
+#define DSIM_MAIN_VRESOL(x) (((x) & 0x7ff) << 16)
+#define DSIM_MAIN_HRESOL(x) (((x) & 0X7ff) << 0)
+
+/* S5P_DSIM_MVPORCH */
+#define DSIM_CMD_ALLOW_SHIFT (28)
+#define DSIM_STABLE_VFP_SHIFT (16)
+#define DSIM_MAIN_VBP_SHIFT (0)
+#define DSIM_CMD_ALLOW_MASK (0xf << DSIM_CMD_ALLOW_SHIFT)
+#define DSIM_STABLE_VFP_MASK (0x7ff << DSIM_STABLE_VFP_SHIFT)
+#define DSIM_MAIN_VBP_MASK (0x7ff << DSIM_MAIN_VBP_SHIFT)
+
+/* S5P_DSIM_MHPORCH */
+#define DSIM_MAIN_HFP_SHIFT (16)
+#define DSIM_MAIN_HBP_SHIFT (0)
+#define DSIM_MAIN_HFP_MASK ((0xffff) << DSIM_MAIN_HFP_SHIFT)
+#define DSIM_MAIN_HBP_MASK ((0xffff) << DSIM_MAIN_HBP_SHIFT)
+
+/* S5P_DSIM_MSYNC */
+#define DSIM_MAIN_VSA_SHIFT (22)
+#define DSIM_MAIN_HSA_SHIFT (0)
+#define DSIM_MAIN_VSA_MASK ((0x3ff) << DSIM_MAIN_VSA_SHIFT)
+#define DSIM_MAIN_HSA_MASK ((0xffff) << DSIM_MAIN_HSA_SHIFT)
+
+/* S5P_DSIM_SDRESOL */
+#define DSIM_SUB_STANDY_SHIFT (31)
+#define DSIM_SUB_VRESOL_SHIFT (16)
+#define DSIM_SUB_HRESOL_SHIFT (0)
+#define DSIM_SUB_STANDY_MASK ((0x1) << DSIM_SUB_STANDY_SHIFT)
+#define DSIM_SUB_VRESOL_MASK ((0x7ff) << DSIM_SUB_VRESOL_SHIFT)
+#define DSIM_SUB_HRESOL_MASK ((0x7ff) << DSIM_SUB_HRESOL_SHIFT)
+
+/* S5P_DSIM_INTSRC */
+#define INTSRC_FRAME_DONE (1 << 24)
+#define INTSRC_PLL_STABLE (1 << 31)
+#define INTSRC_SFR_FIFO_EMPTY (1 << 29)
+
+/* S5P_DSIM_INTMSK */
+#define INTMSK_FRAME_DONE (1 << 24)
+
+/* S5P_DSIM_FIFOCTRL */
+#define SFR_HEADER_EMPTY (1 << 22)
+
+/* S5P_DSIM_PHYACCHR */
+#define DSIM_AFC_CTL(x) (((x) & 0x7) << 5)
+
+/* S5P_DSIM_PLLCTRL */
+#define DSIM_PLL_EN_SHIFT (23)
+#define DSIM_FREQ_BAND_SHIFT (24)
+
+#endif /* _REGS_MIPIDSIM_H */
--- /dev/null
+/* arch/arm/plat-samsung/include/plat/regs-usb3-exynos-drd-phy.h
+ *
+ * Copyright (c) 2011 Samsung Electronics Co. Ltd
+ * Author: Anton Tikhomirov <av.tikhomirov@samsung.com>
+ *
+ * Exynos SuperSpeed USB 3.0 DRD Controller PHY registers
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __PLAT_SAMSUNG_REGS_USB3_EXYNOS_DRD_PHY_H
+#define __PLAT_SAMSUNG_REGS_USB3_EXYNOS_DRD_PHY_H __FILE__
+
+#define EXYNOS_USB3_PHYREG(x) ((x) + S5P_VA_DRD_PHY)
+
+#define EXYNOS_USB3_LINKSYSTEM EXYNOS_USB3_PHYREG(0x04)
+#define EXYNOS_USB3_PHYUTMI EXYNOS_USB3_PHYREG(0x08)
+
+#define EXYNOS_USB3_PHYUTMI_OTGDISABLE (1 << 6)
+#define EXYNOS_USB3_PHYUTMI_FORCESUSPEND (1 << 1)
+#define EXYNOS_USB3_PHYUTMI_FORCESLEEP (1 << 0)
+
+#define EXYNOS_USB3_PHYPIPE EXYNOS_USB3_PHYREG(0x0C)
+#define EXYNOS_USB3_PHYCLKRST EXYNOS_USB3_PHYREG(0x10)
+
+#define EXYNOS_USB3_PHYCLKRST_SSC_REF_CLK_SEL_MASK (0xff << 23)
+#define EXYNOS_USB3_PHYCLKRST_SSC_REF_CLK_SEL_SHIFT (23)
+#define EXYNOS_USB3_PHYCLKRST_SSC_REF_CLK_SEL_LIMIT (0xff)
+#define EXYNOS_USB3_PHYCLKRST_SSC_REF_CLK_SEL(_x) ((_x) << 23)
+
+#define EXYNOS_USB3_PHYCLKRST_SSC_RANGE_MASK (0x03 << 21)
+#define EXYNOS_USB3_PHYCLKRST_SSC_RANGE_SHIFT (21)
+#define EXYNOS_USB3_PHYCLKRST_SSC_RANGE_LIMIT (0x03)
+#define EXYNOS_USB3_PHYCLKRST_SSC_RANGE(_x) ((_x) << 21)
+
+#define EXYNOS_USB3_PHYCLKRST_SSC_EN (1 << 20)
+#define EXYNOS_USB3_PHYCLKRST_REF_SSP_EN (1 << 19)
+#define EXYNOS_USB3_PHYCLKRST_REF_CLKDIV2 (1 << 18)
+
+#define EXYNOS_USB3_PHYCLKRST_MPLL_MULTIPLIER_MASK (0x7f << 11)
+#define EXYNOS_USB3_PHYCLKRST_MPLL_MULTIPLIER_SHIFT (11)
+#define EXYNOS_USB3_PHYCLKRST_MPLL_MULTIPLIER_LIMIT (0x7f)
+#define EXYNOS_USB3_PHYCLKRST_MPLL_MULTIPLIER(_x) ((_x) << 11)
+
+#define EXYNOS_USB3_PHYCLKRST_FSEL_MASK (0x3f << 5)
+#define EXYNOS_USB3_PHYCLKRST_FSEL_SHIFT (5)
+#define EXYNOS_USB3_PHYCLKRST_FSEL_LIMIT (0x3f)
+#define EXYNOS_USB3_PHYCLKRST_FSEL(_x) ((_x) << 5)
+
+#define EXYNOS_USB3_PHYCLKRST_RETENABLEN (1 << 4)
+
+#define EXYNOS_USB3_PHYCLKRST_REFCLKSEL_MASK (0x03 << 2)
+#define EXYNOS_USB3_PHYCLKRST_REFCLKSEL_SHIFT (2)
+#define EXYNOS_USB3_PHYCLKRST_REFCLKSEL_LIMIT (0x03)
+#define EXYNOS_USB3_PHYCLKRST_REFCLKSEL(_x) ((_x) << 2)
+
+#define EXYNOS_USB3_PHYCLKRST_PORTRESET (1 << 1)
+#define EXYNOS_USB3_PHYCLKRST_COMMONONN (1 << 0)
+
+#define EXYNOS_USB3_PHYREG0 EXYNOS_USB3_PHYREG(0x14)
+#define EXYNOS_USB3_PHYREG1 EXYNOS_USB3_PHYREG(0x18)
+#define EXYNOS_USB3_PHYPARAM0 EXYNOS_USB3_PHYREG(0x1C)
+#define EXYNOS_USB3_PHYPARAM1 EXYNOS_USB3_PHYREG(0x20)
+#define EXYNOS_USB3_PHYTERM EXYNOS_USB3_PHYREG(0x24)
+#define EXYNOS_USB3_PHYTEST EXYNOS_USB3_PHYREG(0x28)
+#define EXYNOS_USB3_PHYADP EXYNOS_USB3_PHYREG(0x2C)
+#define EXYNOS_USB3_PHYBATCHG EXYNOS_USB3_PHYREG(0x30)
+#define EXYNOS_USB3_PHYRESUME EXYNOS_USB3_PHYREG(0x34)
+#define EXYNOS_USB3_LINKPORT EXYNOS_USB3_PHYREG(0x44)
+#endif /* __PLAT_SAMSUNG_REGS_USB3_EXYNOS_DRD_PHY_H */
* @fb_delay: Slave specific feedback delay.
* Refer to FB_CLK_SEL register definition in SPI chapter.
* @line: Custom 'identity' of the CS line.
- * @set_level: CS line control.
*
* This is per SPI-Slave Chipselect information.
* Allocate and initialize one in machine init code and make the
struct s3c64xx_spi_csinfo {
u8 fb_delay;
unsigned line;
- void (*set_level)(unsigned line_id, int lvl);
};
/**
* struct s3c64xx_spi_info - SPI Controller defining structure
* @src_clk_nr: Clock source index for the CLK_CFG[SPI_CLKSEL] field.
- * @clk_from_cmu: If the SPI clock/prescalar control block is present
- * by the platform's clock-management-unit and not in SPI controller.
* @num_cs: Number of CS this controller emulates.
* @cfg_gpio: Configure pins for this SPI controller.
- * @fifo_lvl_mask: All tx fifo_lvl fields start at offset-6
- * @rx_lvl_offset: Depends on tx fifo_lvl field and bus number
- * @high_speed: If the controller supports HIGH_SPEED_EN bit
- * @tx_st_done: Depends on tx fifo_lvl field
*/
struct s3c64xx_spi_info {
int src_clk_nr;
- bool clk_from_cmu;
-
int num_cs;
-
- int (*cfg_gpio)(struct platform_device *pdev);
-
- /* Following two fields are for future compatibility */
- int fifo_lvl_mask;
- int rx_lvl_offset;
- int high_speed;
- int tx_st_done;
+ int (*cfg_gpio)(void);
};
/**
* s3c64xx_spi_set_platdata - SPI Controller configure callback by the board
* initialization code.
- * @pd: SPI platform data to set.
+ * @cfg_gpio: Pointer to gpio setup function.
* @src_clk_nr: Clock the SPI controller is to use to generate SPI clocks.
* @num_cs: Number of elements in the 'cs' array.
*
* Call this from machine init code for each SPI Controller that
* has some chips attached to it.
*/
-extern void s3c64xx_spi0_set_platdata(struct s3c64xx_spi_info *pd,
- int src_clk_nr, int num_cs);
-extern void s3c64xx_spi1_set_platdata(struct s3c64xx_spi_info *pd,
- int src_clk_nr, int num_cs);
-extern void s3c64xx_spi2_set_platdata(struct s3c64xx_spi_info *pd,
- int src_clk_nr, int num_cs);
+extern void s3c64xx_spi0_set_platdata(int (*cfg_gpio)(void), int src_clk_nr,
+ int num_cs);
+extern void s3c64xx_spi1_set_platdata(int (*cfg_gpio)(void), int src_clk_nr,
+ int num_cs);
+extern void s3c64xx_spi2_set_platdata(int (*cfg_gpio)(void), int src_clk_nr,
+ int num_cs);
/* defined by architecture to configure gpio */
-extern int s3c64xx_spi0_cfg_gpio(struct platform_device *dev);
-extern int s3c64xx_spi1_cfg_gpio(struct platform_device *dev);
-extern int s3c64xx_spi2_cfg_gpio(struct platform_device *dev);
+extern int s3c64xx_spi0_cfg_gpio(void);
+extern int s3c64xx_spi1_cfg_gpio(void);
+extern int s3c64xx_spi2_cfg_gpio(void);
extern struct s3c64xx_spi_info s3c64xx_spi0_pdata;
extern struct s3c64xx_spi_info s3c64xx_spi1_pdata;
+++ /dev/null
-/* linux/arch/arm/plat-samsung/include/plat/sysmmu.h
- *
- * Copyright (c) 2010-2011 Samsung Electronics Co., Ltd.
- * http://www.samsung.com
- *
- * Samsung System MMU driver for S5P platform
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
-*/
-
-#ifndef __PLAT_SAMSUNG_SYSMMU_H
-#define __PLAT_SAMSUNG_SYSMMU_H __FILE__
-
-enum S5P_SYSMMU_INTERRUPT_TYPE {
- SYSMMU_PAGEFAULT,
- SYSMMU_AR_MULTIHIT,
- SYSMMU_AW_MULTIHIT,
- SYSMMU_BUSERROR,
- SYSMMU_AR_SECURITY,
- SYSMMU_AR_ACCESS,
- SYSMMU_AW_SECURITY,
- SYSMMU_AW_PROTECTION, /* 7 */
- SYSMMU_FAULTS_NUM
-};
-
-#ifdef CONFIG_S5P_SYSTEM_MMU
-
-#include <mach/sysmmu.h>
-
-/**
- * s5p_sysmmu_enable() - enable system mmu of ip
- * @ips: The ip connected system mmu.
- * #pgd: Base physical address of the 1st level page table
- *
- * This function enable system mmu to transfer address
- * from virtual address to physical address
- */
-void s5p_sysmmu_enable(sysmmu_ips ips, unsigned long pgd);
-
-/**
- * s5p_sysmmu_disable() - disable sysmmu mmu of ip
- * @ips: The ip connected system mmu.
- *
- * This function disable system mmu to transfer address
- * from virtual address to physical address
- */
-void s5p_sysmmu_disable(sysmmu_ips ips);
-
-/**
- * s5p_sysmmu_set_tablebase_pgd() - set page table base address to refer page table
- * @ips: The ip connected system mmu.
- * @pgd: The page table base address.
- *
- * This function set page table base address
- * When system mmu transfer address from virtaul address to physical address,
- * system mmu refer address information from page table
- */
-void s5p_sysmmu_set_tablebase_pgd(sysmmu_ips ips, unsigned long pgd);
-
-/**
- * s5p_sysmmu_tlb_invalidate() - flush all TLB entry in system mmu
- * @ips: The ip connected system mmu.
- *
- * This function flush all TLB entry in system mmu
- */
-void s5p_sysmmu_tlb_invalidate(sysmmu_ips ips);
-
-/** s5p_sysmmu_set_fault_handler() - Fault handler for System MMUs
- * @itype: type of fault.
- * @pgtable_base: the physical address of page table base. This is 0 if @ips is
- * SYSMMU_BUSERROR.
- * @fault_addr: the device (virtual) address that the System MMU tried to
- * translated. This is 0 if @ips is SYSMMU_BUSERROR.
- * Called when interrupt occurred by the System MMUs
- * The device drivers of peripheral devices that has a System MMU can implement
- * a fault handler to resolve address translation fault by System MMU.
- * The meanings of return value and parameters are described below.
-
- * return value: non-zero if the fault is correctly resolved.
- * zero if the fault is not handled.
- */
-void s5p_sysmmu_set_fault_handler(sysmmu_ips ips,
- int (*handler)(enum S5P_SYSMMU_INTERRUPT_TYPE itype,
- unsigned long pgtable_base,
- unsigned long fault_addr));
-#else
-#define s5p_sysmmu_enable(ips, pgd) do { } while (0)
-#define s5p_sysmmu_disable(ips) do { } while (0)
-#define s5p_sysmmu_set_tablebase_pgd(ips, pgd) do { } while (0)
-#define s5p_sysmmu_tlb_invalidate(ips) do { } while (0)
-#define s5p_sysmmu_set_fault_handler(ips, handler) do { } while (0)
-#endif
-#endif /* __ASM_PLAT_SYSMMU_H */
#endif
}
+extern void s5p_v4l2_int_src_hdmi_hpd(void);
+extern void s5p_int_src_hdmi_hpd(struct platform_device *pdev);
+extern int s5p_v4l2_hpd_read_gpio(void);
+extern void s5p_v4l2_int_src_ext_hpd(void);
+extern void s5p_cec_cfg_gpio(struct platform_device *pdev);
+extern void s5p_v4l2_int_src_hdmi_hpd(void);
#endif /* __SAMSUNG_PLAT_TV_H */
enum s5p_usb_phy_type {
S5P_USB_PHY_DEVICE,
S5P_USB_PHY_HOST,
+ S5P_USB_PHY_DRD,
};
extern int s5p_usb_phy_init(struct platform_device *pdev, int type);
select ARCH_WANT_OPTIONAL_GPIOLIB
select ARCH_WANT_FRAME_POINTERS
select HAVE_DMA_ATTRS
+ select HAVE_DMA_CONTIGUOUS if !SWIOTLB
select HAVE_KRETPROBES
select HAVE_OPTPROBES
select HAVE_FTRACE_MCOUNT_RECORD
--- /dev/null
+#ifndef ASMX86_DMA_CONTIGUOUS_H
+#define ASMX86_DMA_CONTIGUOUS_H
+
+#ifdef __KERNEL__
+
+#include <linux/types.h>
+#include <asm-generic/dma-contiguous.h>
+
+static inline void
+dma_contiguous_early_fixup(phys_addr_t base, unsigned long size) { }
+
+#endif
+#endif
#include <asm/io.h>
#include <asm/swiotlb.h>
#include <asm-generic/dma-coherent.h>
+#include <linux/dma-contiguous.h>
#ifdef CONFIG_ISA
# define ISA_DMA_BIT_MASK DMA_BIT_MASK(24)
dma_addr_t *dma_addr, gfp_t flag,
struct dma_attrs *attrs);
+extern void dma_generic_free_coherent(struct device *dev, size_t size,
+ void *vaddr, dma_addr_t dma_addr,
+ struct dma_attrs *attrs);
+
static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size)
{
if (!dev->dma_mask)
struct dma_attrs *attrs)
{
unsigned long dma_mask;
- struct page *page;
+ struct page *page = NULL;
+ unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT;
dma_addr_t addr;
dma_mask = dma_alloc_coherent_mask(dev, flag);
flag |= __GFP_ZERO;
again:
- page = alloc_pages_node(dev_to_node(dev), flag, get_order(size));
+ if (!(flag & GFP_ATOMIC))
+ page = dma_alloc_from_contiguous(dev, count, get_order(size));
+ if (!page)
+ page = alloc_pages_node(dev_to_node(dev), flag, get_order(size));
if (!page)
return NULL;
return page_address(page);
}
+void dma_generic_free_coherent(struct device *dev, size_t size, void *vaddr,
+ dma_addr_t dma_addr, struct dma_attrs *attrs)
+{
+ unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT;
+ struct page *page = virt_to_page(vaddr);
+
+ if (!dma_release_from_contiguous(dev, page, count))
+ free_pages((unsigned long)vaddr, get_order(size));
+}
+
/*
* See <Documentation/x86/x86_64/boot-options.txt> for the iommu kernel
* parameter documentation.
return nents;
}
-static void nommu_free_coherent(struct device *dev, size_t size, void *vaddr,
- dma_addr_t dma_addr, struct dma_attrs *attrs)
-{
- free_pages((unsigned long)vaddr, get_order(size));
-}
-
static void nommu_sync_single_for_device(struct device *dev,
dma_addr_t addr, size_t size,
enum dma_data_direction dir)
struct dma_map_ops nommu_dma_ops = {
.alloc = dma_generic_alloc_coherent,
- .free = nommu_free_coherent,
+ .free = dma_generic_free_coherent,
.map_sg = nommu_map_sg,
.map_page = nommu_map_page,
.sync_single_for_device = nommu_sync_single_for_device,
#include <asm/pci-direct.h>
#include <linux/init_ohci1394_dma.h>
#include <linux/kvm_para.h>
+#include <linux/dma-contiguous.h>
#include <linux/errno.h>
#include <linux/kernel.h>
}
#endif
memblock.current_limit = get_max_mapped();
+ dma_contiguous_reserve(0);
/*
* NOTE: On x86-32, only from this point on, fixmaps are ready for use.
source "drivers/devfreq/Kconfig"
+source "drivers/gpu/vithar/Kconfig"
+
endmenu
APIs extension; the file's descriptor can then be passed on to other
driver.
+config CMA
+ bool "Contiguous Memory Allocator (EXPERIMENTAL)"
+ depends on HAVE_DMA_CONTIGUOUS && HAVE_MEMBLOCK && EXPERIMENTAL
+ select MIGRATION
+ help
+ This enables the Contiguous Memory Allocator which allows drivers
+ to allocate big physically-contiguous blocks of memory for use with
+ hardware components that do not support I/O map nor scatter-gather.
+
+ For more information see <include/linux/dma-contiguous.h>.
+ If unsure, say "n".
+
+if CMA
+
+config CMA_DEBUG
+ bool "CMA debug messages (DEVELOPMENT)"
+ depends on DEBUG_KERNEL
+ help
+ Turns on debug messages in CMA. This produces KERN_DEBUG
+ messages for every CMA call as well as various messages while
+ processing calls such as dma_alloc_from_contiguous().
+ This option does not affect warning and error messages.
+
+comment "Default contiguous memory area size:"
+
+config CMA_SIZE_MBYTES
+ int "Size in Mega Bytes"
+ depends on !CMA_SIZE_SEL_PERCENTAGE
+ default 16
+ help
+ Defines the size (in MiB) of the default memory area for Contiguous
+ Memory Allocator.
+
+config CMA_SIZE_PERCENTAGE
+ int "Percentage of total memory"
+ depends on !CMA_SIZE_SEL_MBYTES
+ default 10
+ help
+ Defines the size of the default memory area for Contiguous Memory
+ Allocator as a percentage of the total memory in the system.
+
+choice
+ prompt "Selected region size"
+ default CMA_SIZE_SEL_ABSOLUTE
+
+config CMA_SIZE_SEL_MBYTES
+ bool "Use mega bytes value only"
+
+config CMA_SIZE_SEL_PERCENTAGE
+ bool "Use percentage value only"
+
+config CMA_SIZE_SEL_MIN
+ bool "Use lower value (minimum)"
+
+config CMA_SIZE_SEL_MAX
+ bool "Use higher value (maximum)"
+
+endchoice
+
+config CMA_ALIGNMENT
+ int "Maximum PAGE_SIZE order of alignment for contiguous buffers"
+ range 4 9
+ default 8
+ help
+ DMA mapping framework by default aligns all buffers to the smallest
+ PAGE_SIZE order which is greater than or equal to the requested buffer
+ size. This works well for buffers up to a few hundreds kilobytes, but
+ for larger buffers it just a memory waste. With this parameter you can
+ specify the maximum PAGE_SIZE order for contiguous buffers. Larger
+ buffers will be aligned only to this specified order. The order is
+ expressed as a power of two multiplied by the PAGE_SIZE.
+
+ For example, if your system defaults to 4KiB pages, the order value
+ of 8 means that the buffers will be aligned up to 1MiB only.
+
+ If unsure, leave the default value "8".
+
+config CMA_AREAS
+ int "Maximum count of the CMA device-private areas"
+ default 7
+ help
+ CMA allows to create CMA areas for particular devices. This parameter
+ sets the maximum number of such device private CMA areas in the
+ system.
+
+ If unsure, leave the default value "7".
+
+endif
+
endmenu
attribute_container.o transport_class.o \
topology.o
obj-$(CONFIG_DEVTMPFS) += devtmpfs.o
+obj-$(CONFIG_CMA) += dma-contiguous.o
obj-y += power/
obj-$(CONFIG_HAS_DMA) += dma-mapping.o
obj-$(CONFIG_HAVE_GENERIC_DMA_COHERENT) += dma-coherent.o
return 0;
}
+static int dma_buf_mmap_internal(struct file *file, struct vm_area_struct *vma)
+{
+ struct dma_buf *dmabuf;
+
+ if (!is_dma_buf_file(file))
+ return -EINVAL;
+
+ dmabuf = file->private_data;
+
+ /* check for overflowing the buffer's size */
+ if (vma->vm_pgoff + ((vma->vm_end - vma->vm_start) >> PAGE_SHIFT) >
+ dmabuf->size >> PAGE_SHIFT)
+ return -EINVAL;
+
+ return dmabuf->ops->mmap(dmabuf, vma);
+}
+
static const struct file_operations dma_buf_fops = {
.release = dma_buf_release,
+ .mmap = dma_buf_mmap_internal,
};
/*
|| !ops->unmap_dma_buf
|| !ops->release
|| !ops->kmap_atomic
- || !ops->kmap)) {
+ || !ops->kmap
+ || !ops->mmap)) {
return ERR_PTR(-EINVAL);
}
dmabuf->ops->kunmap(dmabuf, page_num, vaddr);
}
EXPORT_SYMBOL_GPL(dma_buf_kunmap);
+
+
+/**
+ * dma_buf_mmap - Setup up a userspace mmap with the given vma
+ * @dmabuf: [in] buffer that should back the vma
+ * @vma: [in] vma for the mmap
+ * @pgoff: [in] offset in pages where this mmap should start within the
+ * dma-buf buffer.
+ *
+ * This function adjusts the passed in vma so that it points at the file of the
+ * dma_buf operation. It alsog adjusts the starting pgoff and does bounds
+ * checking on the size of the vma. Then it calls the exporters mmap function to
+ * set up the mapping.
+ *
+ * Can return negative error values, returns 0 on success.
+ */
+int dma_buf_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma,
+ unsigned long pgoff)
+{
+ if (WARN_ON(!dmabuf || !vma))
+ return -EINVAL;
+
+ /* check for offset overflow */
+ if (pgoff + ((vma->vm_end - vma->vm_start) >> PAGE_SHIFT) < pgoff)
+ return -EOVERFLOW;
+
+ /* check for overflowing the buffer's size */
+ if (pgoff + ((vma->vm_end - vma->vm_start) >> PAGE_SHIFT) >
+ dmabuf->size >> PAGE_SHIFT)
+ return -EINVAL;
+
+ /* readjust the vma */
+ if (vma->vm_file)
+ fput(vma->vm_file);
+
+ vma->vm_file = dmabuf->file;
+ get_file(vma->vm_file);
+
+ vma->vm_pgoff = pgoff;
+
+ return dmabuf->ops->mmap(dmabuf, vma);
+}
+EXPORT_SYMBOL_GPL(dma_buf_mmap);
+
+/**
+ * dma_buf_vmap - Create virtual mapping for the buffer object into kernel
+ * address space. Same restrictions as for vmap and friends apply.
+ * @dmabuf: [in] buffer to vmap
+ *
+ * This call may fail due to lack of virtual mapping address space.
+ * These calls are optional in drivers. The intended use for them
+ * is for mapping objects linear in kernel space for high use objects.
+ * Please attempt to use kmap/kunmap before thinking about these interfaces.
+ */
+void *dma_buf_vmap(struct dma_buf *dmabuf)
+{
+ if (WARN_ON(!dmabuf))
+ return NULL;
+
+ if (dmabuf->ops->vmap)
+ return dmabuf->ops->vmap(dmabuf);
+ return NULL;
+}
+EXPORT_SYMBOL_GPL(dma_buf_vmap);
+
+/**
+ * dma_buf_vunmap - Unmap a vmap obtained by dma_buf_vmap.
+ * @dmabuf: [in] buffer to vunmap
+ */
+void dma_buf_vunmap(struct dma_buf *dmabuf, void *vaddr)
+{
+ if (WARN_ON(!dmabuf))
+ return;
+
+ if (dmabuf->ops->vunmap)
+ dmabuf->ops->vunmap(dmabuf, vaddr);
+}
+EXPORT_SYMBOL_GPL(dma_buf_vunmap);
struct dma_coherent_mem {
void *virt_base;
dma_addr_t device_base;
+ phys_addr_t pfn_base;
int size;
int flags;
unsigned long *bitmap;
dev->dma_mem->virt_base = mem_base;
dev->dma_mem->device_base = device_addr;
+ dev->dma_mem->pfn_base = PFN_DOWN(bus_addr);
dev->dma_mem->size = pages;
dev->dma_mem->flags = flags;
return 0;
}
EXPORT_SYMBOL(dma_release_from_coherent);
+
+/**
+ * dma_mmap_from_coherent() - try to mmap the memory allocated from
+ * per-device coherent memory pool to userspace
+ * @dev: device from which the memory was allocated
+ * @vma: vm_area for the userspace memory
+ * @vaddr: cpu address returned by dma_alloc_from_coherent
+ * @size: size of the memory buffer allocated by dma_alloc_from_coherent
+ *
+ * This checks whether the memory was allocated from the per-device
+ * coherent memory pool and if so, maps that memory to the provided vma.
+ *
+ * Returns 1 if we correctly mapped the memory, or 0 if
+ * dma_release_coherent() should proceed with mapping memory from
+ * generic pools.
+ */
+int dma_mmap_from_coherent(struct device *dev, struct vm_area_struct *vma,
+ void *vaddr, size_t size, int *ret)
+{
+ struct dma_coherent_mem *mem = dev ? dev->dma_mem : NULL;
+
+ if (mem && vaddr >= mem->virt_base && vaddr + size <=
+ (mem->virt_base + (mem->size << PAGE_SHIFT))) {
+ unsigned long off = vma->vm_pgoff;
+ int start = (vaddr - mem->virt_base) >> PAGE_SHIFT;
+ int user_count = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT;
+ int count = size >> PAGE_SHIFT;
+
+ *ret = -ENXIO;
+ if (off < count && user_count <= count - off) {
+ unsigned pfn = mem->pfn_base + start + off;
+ *ret = remap_pfn_range(vma, vma->vm_start, pfn,
+ user_count << PAGE_SHIFT,
+ vma->vm_page_prot);
+ }
+ return 1;
+ }
+ return 0;
+}
+EXPORT_SYMBOL(dma_mmap_from_coherent);
--- /dev/null
+/*
+ * Contiguous Memory Allocator for DMA mapping framework
+ * Copyright (c) 2010-2011 by Samsung Electronics.
+ * Written by:
+ * Marek Szyprowski <m.szyprowski@samsung.com>
+ * Michal Nazarewicz <mina86@mina86.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of the
+ * License or (at your optional) any later version of the license.
+ */
+
+#define pr_fmt(fmt) "cma: " fmt
+
+#ifdef CONFIG_CMA_DEBUG
+#ifndef DEBUG
+# define DEBUG
+#endif
+#endif
+
+#include <asm/page.h>
+#include <asm/dma-contiguous.h>
+
+#include <linux/memblock.h>
+#include <linux/err.h>
+#include <linux/mm.h>
+#include <linux/mutex.h>
+#include <linux/page-isolation.h>
+#include <linux/slab.h>
+#include <linux/swap.h>
+#include <linux/mm_types.h>
+#include <linux/dma-contiguous.h>
+
+#ifndef SZ_1M
+#define SZ_1M (1 << 20)
+#endif
+
+struct cma {
+ unsigned long base_pfn;
+ unsigned long count;
+ unsigned long *bitmap;
+};
+
+struct cma *dma_contiguous_default_area;
+
+#ifdef CONFIG_CMA_SIZE_MBYTES
+#define CMA_SIZE_MBYTES CONFIG_CMA_SIZE_MBYTES
+#else
+#define CMA_SIZE_MBYTES 0
+#endif
+
+/*
+ * Default global CMA area size can be defined in kernel's .config.
+ * This is usefull mainly for distro maintainers to create a kernel
+ * that works correctly for most supported systems.
+ * The size can be set in bytes or as a percentage of the total memory
+ * in the system.
+ *
+ * Users, who want to set the size of global CMA area for their system
+ * should use cma= kernel parameter.
+ */
+static const unsigned long size_bytes = CMA_SIZE_MBYTES * SZ_1M;
+static long size_cmdline = -1;
+
+static int __init early_cma(char *p)
+{
+ pr_debug("%s(%s)\n", __func__, p);
+ size_cmdline = memparse(p, &p);
+ return 0;
+}
+early_param("cma", early_cma);
+
+#ifdef CONFIG_CMA_SIZE_PERCENTAGE
+
+static unsigned long __init __maybe_unused cma_early_percent_memory(void)
+{
+ struct memblock_region *reg;
+ unsigned long total_pages = 0;
+
+ /*
+ * We cannot use memblock_phys_mem_size() here, because
+ * memblock_analyze() has not been called yet.
+ */
+ for_each_memblock(memory, reg)
+ total_pages += memblock_region_memory_end_pfn(reg) -
+ memblock_region_memory_base_pfn(reg);
+
+ return (total_pages * CONFIG_CMA_SIZE_PERCENTAGE / 100) << PAGE_SHIFT;
+}
+
+#else
+
+static inline __maybe_unused unsigned long cma_early_percent_memory(void)
+{
+ return 0;
+}
+
+#endif
+
+/**
+ * dma_contiguous_reserve() - reserve area for contiguous memory handling
+ * @limit: End address of the reserved memory (optional, 0 for any).
+ *
+ * This function reserves memory from early allocator. It should be
+ * called by arch specific code once the early allocator (memblock or bootmem)
+ * has been activated and all other subsystems have already allocated/reserved
+ * memory.
+ */
+void __init dma_contiguous_reserve(phys_addr_t limit)
+{
+ unsigned long selected_size = 0;
+
+ pr_debug("%s(limit %08lx)\n", __func__, (unsigned long)limit);
+
+ if (size_cmdline != -1) {
+ selected_size = size_cmdline;
+ } else {
+#ifdef CONFIG_CMA_SIZE_SEL_MBYTES
+ selected_size = size_bytes;
+#elif defined(CONFIG_CMA_SIZE_SEL_PERCENTAGE)
+ selected_size = cma_early_percent_memory();
+#elif defined(CONFIG_CMA_SIZE_SEL_MIN)
+ selected_size = min(size_bytes, cma_early_percent_memory());
+#elif defined(CONFIG_CMA_SIZE_SEL_MAX)
+ selected_size = max(size_bytes, cma_early_percent_memory());
+#endif
+ }
+
+ if (selected_size) {
+ pr_debug("%s: reserving %ld MiB for global area\n", __func__,
+ selected_size / SZ_1M);
+
+ dma_declare_contiguous(NULL, selected_size, 0, limit);
+ }
+};
+
+static DEFINE_MUTEX(cma_mutex);
+
+static __init int cma_activate_area(unsigned long base_pfn, unsigned long count)
+{
+ unsigned long pfn = base_pfn;
+ unsigned i = count >> pageblock_order;
+ struct zone *zone;
+
+ WARN_ON_ONCE(!pfn_valid(pfn));
+ zone = page_zone(pfn_to_page(pfn));
+
+ do {
+ unsigned j;
+ base_pfn = pfn;
+ for (j = pageblock_nr_pages; j; --j, pfn++) {
+ WARN_ON_ONCE(!pfn_valid(pfn));
+ if (page_zone(pfn_to_page(pfn)) != zone)
+ return -EINVAL;
+ }
+ init_cma_reserved_pageblock(pfn_to_page(base_pfn));
+ } while (--i);
+ return 0;
+}
+
+static __init struct cma *cma_create_area(unsigned long base_pfn,
+ unsigned long count)
+{
+ int bitmap_size = BITS_TO_LONGS(count) * sizeof(long);
+ struct cma *cma;
+ int ret = -ENOMEM;
+
+ pr_debug("%s(base %08lx, count %lx)\n", __func__, base_pfn, count);
+
+ cma = kmalloc(sizeof *cma, GFP_KERNEL);
+ if (!cma)
+ return ERR_PTR(-ENOMEM);
+
+ cma->base_pfn = base_pfn;
+ cma->count = count;
+ cma->bitmap = kzalloc(bitmap_size, GFP_KERNEL);
+
+ if (!cma->bitmap)
+ goto no_mem;
+
+ ret = cma_activate_area(base_pfn, count);
+ if (ret)
+ goto error;
+
+ pr_debug("%s: returned %p\n", __func__, (void *)cma);
+ return cma;
+
+error:
+ kfree(cma->bitmap);
+no_mem:
+ kfree(cma);
+ return ERR_PTR(ret);
+}
+
+static struct cma_reserved {
+ phys_addr_t start;
+ unsigned long size;
+ struct device *dev;
+} cma_reserved[MAX_CMA_AREAS] __initdata;
+static unsigned cma_reserved_count __initdata;
+
+static int __init cma_init_reserved_areas(void)
+{
+ struct cma_reserved *r = cma_reserved;
+ unsigned i = cma_reserved_count;
+
+ pr_debug("%s()\n", __func__);
+
+ for (; i; --i, ++r) {
+ struct cma *cma;
+ cma = cma_create_area(PFN_DOWN(r->start),
+ r->size >> PAGE_SHIFT);
+ if (!IS_ERR(cma))
+ dev_set_cma_area(r->dev, cma);
+ }
+ return 0;
+}
+core_initcall(cma_init_reserved_areas);
+
+/**
+ * dma_declare_contiguous() - reserve area for contiguous memory handling
+ * for particular device
+ * @dev: Pointer to device structure.
+ * @size: Size of the reserved memory.
+ * @base: Start address of the reserved memory (optional, 0 for any).
+ * @limit: End address of the reserved memory (optional, 0 for any).
+ *
+ * This function reserves memory for specified device. It should be
+ * called by board specific code when early allocator (memblock or bootmem)
+ * is still activate.
+ */
+int __init dma_declare_contiguous(struct device *dev, unsigned long size,
+ phys_addr_t base, phys_addr_t limit)
+{
+ struct cma_reserved *r = &cma_reserved[cma_reserved_count];
+ unsigned long alignment;
+
+ pr_debug("%s(size %lx, base %08lx, limit %08lx)\n", __func__,
+ (unsigned long)size, (unsigned long)base,
+ (unsigned long)limit);
+
+ /* Sanity checks */
+ if (cma_reserved_count == ARRAY_SIZE(cma_reserved)) {
+ pr_err("Not enough slots for CMA reserved regions!\n");
+ return -ENOSPC;
+ }
+
+ if (!size)
+ return -EINVAL;
+
+ /* Sanitise input arguments */
+ alignment = PAGE_SIZE << max(MAX_ORDER, pageblock_order);
+ base = ALIGN(base, alignment);
+ size = ALIGN(size, alignment);
+ limit &= ~(alignment - 1);
+
+ /* Reserve memory */
+ if (base) {
+ if (memblock_is_region_reserved(base, size) ||
+ memblock_reserve(base, size) < 0) {
+ base = -EBUSY;
+ goto err;
+ }
+ } else {
+ /*
+ * Use __memblock_alloc_base() since
+ * memblock_alloc_base() panic()s.
+ */
+ phys_addr_t addr = __memblock_alloc_base(size, alignment, limit);
+ if (!addr) {
+ base = -ENOMEM;
+ goto err;
+ } else if (addr + size > ~(unsigned long)0) {
+ memblock_free(addr, size);
+ base = -EINVAL;
+ goto err;
+ } else {
+ base = addr;
+ }
+ }
+
+ /*
+ * Each reserved area must be initialised later, when more kernel
+ * subsystems (like slab allocator) are available.
+ */
+ r->start = base;
+ r->size = size;
+ r->dev = dev;
+ cma_reserved_count++;
+ pr_info("CMA: reserved %ld MiB at %08lx\n", size / SZ_1M,
+ (unsigned long)base);
+
+ /* Architecture specific contiguous memory fixup. */
+ dma_contiguous_early_fixup(base, size);
+ return 0;
+err:
+ pr_err("CMA: failed to reserve %ld MiB\n", size / SZ_1M);
+ return base;
+}
+
+/**
+ * dma_alloc_from_contiguous() - allocate pages from contiguous area
+ * @dev: Pointer to device for which the allocation is performed.
+ * @count: Requested number of pages.
+ * @align: Requested alignment of pages (in PAGE_SIZE order).
+ *
+ * This function allocates memory buffer for specified device. It uses
+ * device specific contiguous memory area if available or the default
+ * global one. Requires architecture specific get_dev_cma_area() helper
+ * function.
+ */
+struct page *dma_alloc_from_contiguous(struct device *dev, int count,
+ unsigned int align)
+{
+ unsigned long mask, pfn, pageno, start = 0;
+ struct cma *cma = dev_get_cma_area(dev);
+ int ret;
+
+ if (!cma || !cma->count)
+ return NULL;
+
+ if (align > CONFIG_CMA_ALIGNMENT)
+ align = CONFIG_CMA_ALIGNMENT;
+
+ pr_debug("%s(cma %p, count %d, align %d)\n", __func__, (void *)cma,
+ count, align);
+
+ if (!count)
+ return NULL;
+
+ mask = (1 << align) - 1;
+
+ mutex_lock(&cma_mutex);
+
+ for (;;) {
+ pageno = bitmap_find_next_zero_area(cma->bitmap, cma->count,
+ start, count, mask);
+ if (pageno >= cma->count) {
+ ret = -ENOMEM;
+ goto error;
+ }
+
+ pfn = cma->base_pfn + pageno;
+ ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA);
+ if (ret == 0) {
+ bitmap_set(cma->bitmap, pageno, count);
+ break;
+ } else if (ret != -EBUSY) {
+ goto error;
+ }
+ pr_debug("%s(): memory range at %p is busy, retrying\n",
+ __func__, pfn_to_page(pfn));
+ /* try again with a bit different memory target */
+ start = pageno + mask + 1;
+ }
+
+ mutex_unlock(&cma_mutex);
+
+ pr_debug("%s(): returned %p\n", __func__, pfn_to_page(pfn));
+ return pfn_to_page(pfn);
+error:
+ mutex_unlock(&cma_mutex);
+ return NULL;
+}
+
+/**
+ * dma_release_from_contiguous() - release allocated pages
+ * @dev: Pointer to device for which the pages were allocated.
+ * @pages: Allocated pages.
+ * @count: Number of allocated pages.
+ *
+ * This function releases memory allocated by dma_alloc_from_contiguous().
+ * It returns false when provided pages do not belong to contiguous area and
+ * true otherwise.
+ */
+bool dma_release_from_contiguous(struct device *dev, struct page *pages,
+ int count)
+{
+ struct cma *cma = dev_get_cma_area(dev);
+ unsigned long pfn;
+
+ if (!cma || !pages)
+ return false;
+
+ pr_debug("%s(page %p)\n", __func__, (void *)pages);
+
+ pfn = page_to_pfn(pages);
+
+ if (pfn < cma->base_pfn || pfn >= cma->base_pfn + cma->count)
+ return false;
+
+ VM_BUG_ON(pfn + count > cma->base_pfn + cma->count);
+
+ mutex_lock(&cma_mutex);
+ bitmap_clear(cma->bitmap, pfn - cma->base_pfn, count);
+ free_contig_range(pfn, count);
+ mutex_unlock(&cma_mutex);
+
+ return true;
+}
}
EXPORT_SYMBOL(dmam_release_declared_memory);
+/*
+ * Create scatter-list for the already allocated DMA buffer.
+ */
+int dma_common_get_sgtable(struct device *dev, struct sg_table *sgt,
+ void *cpu_addr, dma_addr_t handle, size_t size)
+{
+ struct page *page = virt_to_page(cpu_addr);
+ int ret;
+
+ ret = sg_alloc_table(sgt, 1, GFP_KERNEL);
+ if (unlikely(ret))
+ return ret;
+
+ sg_set_page(sgt->sgl, page, PAGE_ALIGN(size), 0);
+ return 0;
+}
+EXPORT_SYMBOL(dma_common_get_sgtable);
+
#endif
dma_async_tx_callback callback;
/* Change status to reload it */
+ dma_cookie_assign(&desc->txd);
desc->status = PREP;
pch = desc->pchan;
callback = desc->txd.callback;
-obj-y += drm/ vga/ stub/
+obj-y += drm/ vga/ stub/ vithar/
Choose this option if you have a Samsung SoC EXYNOS chipset.
If M is selected the module will be called exynosdrm.
+config DRM_EXYNOS_DMABUF
+ bool "EXYNOS DRM DMABUF"
+ depends on DRM_EXYNOS
+ help
+ Choose this option if you want to use DMABUF feature for DRM.
+
config DRM_EXYNOS_FIMD
bool "Exynos DRM FIMD"
depends on DRM_EXYNOS && !FB_S3C
exynos_drm_buf.o exynos_drm_gem.o exynos_drm_core.o \
exynos_drm_plane.o
+exynosdrm-$(CONFIG_DRM_EXYNOS_DMABUF) += exynos_drm_dmabuf.o
exynosdrm-$(CONFIG_DRM_EXYNOS_FIMD) += exynos_drm_fimd.o
exynosdrm-$(CONFIG_DRM_EXYNOS_HDMI) += exynos_hdmi.o exynos_mixer.o \
exynos_ddc.o exynos_hdmiphy.o \
unsigned int flags, struct exynos_drm_gem_buf *buf)
{
dma_addr_t start_addr;
- unsigned int npages, page_size, i = 0;
+ unsigned int npages, i = 0;
struct scatterlist *sgl;
int ret = 0;
if (buf->size >= SZ_1M) {
npages = buf->size >> SECTION_SHIFT;
- page_size = SECTION_SIZE;
+ buf->page_size = SECTION_SIZE;
} else if (buf->size >= SZ_64K) {
npages = buf->size >> 16;
- page_size = SZ_64K;
+ buf->page_size = SZ_64K;
} else {
npages = buf->size >> PAGE_SHIFT;
- page_size = PAGE_SIZE;
+ buf->page_size = PAGE_SIZE;
}
buf->sgt = kzalloc(sizeof(struct sg_table), GFP_KERNEL);
while (i < npages) {
buf->pages[i] = phys_to_page(start_addr);
- sg_set_page(sgl, buf->pages[i], page_size, 0);
+ sg_set_page(sgl, buf->pages[i], buf->page_size, 0);
sg_dma_address(sgl) = start_addr;
- start_addr += page_size;
+ start_addr += buf->page_size;
sgl = sg_next(sgl);
i++;
}
static LIST_HEAD(exynos_drm_subdrv_list);
static struct drm_device *drm_dev;
+#ifdef CONFIG_EXYNOS_IOMMU
+struct dma_iommu_mapping *exynos_drm_common_mapping;
+#endif
static int exynos_drm_subdrv_probe(struct drm_device *dev,
struct exynos_drm_subdrv *subdrv)
--- /dev/null
+/* exynos_drm_dmabuf.c
+ *
+ * Copyright (c) 2012 Samsung Electronics Co., Ltd.
+ * Author: Inki Dae <inki.dae@samsung.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * VA LINUX SYSTEMS AND/OR ITS SUPPLIERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#include "drmP.h"
+#include "drm.h"
+#include "exynos_drm_drv.h"
+#include "exynos_drm_gem.h"
+
+#include "exynos_drm.h"
+#include <linux/dma-buf.h>
+
+static struct sg_table *exynos_pages_to_sg(struct page **pages, int nr_pages,
+ unsigned int page_size)
+{
+ struct sg_table *sgt = NULL;
+ struct scatterlist *sgl;
+ int i, ret;
+
+ sgt = kzalloc(sizeof(*sgt), GFP_KERNEL);
+ if (!sgt)
+ goto out;
+
+ ret = sg_alloc_table(sgt, nr_pages, GFP_KERNEL);
+ if (ret)
+ goto err_free_sgt;
+
+ if (page_size < PAGE_SIZE)
+ page_size = PAGE_SIZE;
+
+ for_each_sg(sgt->sgl, sgl, nr_pages, i)
+ sg_set_page(sgl, pages[i], page_size, 0);
+
+ return sgt;
+
+err_free_sgt:
+ kfree(sgt);
+ sgt = NULL;
+out:
+ return NULL;
+}
+
+static struct sg_table *
+ exynos_gem_map_dma_buf(struct dma_buf_attachment *attach,
+ enum dma_data_direction dir)
+{
+ struct exynos_drm_gem_obj *gem_obj = attach->dmabuf->priv;
+ struct drm_device *dev = gem_obj->base.dev;
+ struct exynos_drm_gem_buf *buf;
+ struct sg_table *sgt = NULL;
+ unsigned int npages;
+ int nents;
+
+ DRM_DEBUG_PRIME("%s\n", __FILE__);
+
+ mutex_lock(&dev->struct_mutex);
+
+ buf = gem_obj->buffer;
+
+ /* there should always be pages allocated. */
+ if (!buf->pages) {
+ DRM_ERROR("pages is null.\n");
+ goto err_unlock;
+ }
+
+ npages = buf->size / buf->page_size;
+
+ sgt = exynos_pages_to_sg(buf->pages, npages, buf->page_size);
+ nents = dma_map_sg(attach->dev, sgt->sgl, sgt->nents, dir);
+
+ DRM_DEBUG_PRIME("npages = %d buffer size = 0x%lx page_size = 0x%lx\n",
+ npages, buf->size, buf->page_size);
+
+err_unlock:
+ mutex_unlock(&dev->struct_mutex);
+ return sgt;
+}
+
+static void exynos_gem_unmap_dma_buf(struct dma_buf_attachment *attach,
+ struct sg_table *sgt,
+ enum dma_data_direction dir)
+{
+ dma_unmap_sg(attach->dev, sgt->sgl, sgt->nents, dir);
+ sg_free_table(sgt);
+ kfree(sgt);
+ sgt = NULL;
+}
+
+static void exynos_dmabuf_release(struct dma_buf *dmabuf)
+{
+ struct exynos_drm_gem_obj *exynos_gem_obj = dmabuf->priv;
+
+ DRM_DEBUG_PRIME("%s\n", __FILE__);
+
+ /*
+ * exynos_dmabuf_release() call means that file object's
+ * f_count is 0 and it calls drm_gem_object_handle_unreference()
+ * to drop the references that these values had been increased
+ * at drm_prime_handle_to_fd()
+ */
+ if (exynos_gem_obj->base.export_dma_buf == dmabuf) {
+ exynos_gem_obj->base.export_dma_buf = NULL;
+
+ /*
+ * drop this gem object refcount to release allocated buffer
+ * and resources.
+ */
+ drm_gem_object_unreference_unlocked(&exynos_gem_obj->base);
+ }
+}
+
+static void *exynos_gem_dmabuf_kmap_atomic(struct dma_buf *dma_buf,
+ unsigned long page_num)
+{
+ /* TODO */
+
+ return NULL;
+}
+
+static void exynos_gem_dmabuf_kunmap_atomic(struct dma_buf *dma_buf,
+ unsigned long page_num,
+ void *addr)
+{
+ /* TODO */
+}
+
+static void *exynos_gem_dmabuf_kmap(struct dma_buf *dma_buf,
+ unsigned long page_num)
+{
+ /* TODO */
+
+ return NULL;
+}
+
+static void exynos_gem_dmabuf_kunmap(struct dma_buf *dma_buf,
+ unsigned long page_num, void *addr)
+{
+ /* TODO */
+}
+
+static int exynos_drm_gem_dmabuf_mmap(struct dma_buf *dmabuf,
+ struct vm_area_struct *vma)
+{
+ struct drm_gem_object *obj = dmabuf->priv;
+ struct exynos_drm_gem_obj *exynos_gem_obj = to_exynos_gem_obj(obj);
+ struct exynos_drm_gem_buf *buffer = exynos_gem_obj->buffer;
+ unsigned long uaddr = vma->vm_start;
+ int ret;
+
+ if (exynos_gem_obj->flags & EXYNOS_BO_NONCONTIG) {
+ unsigned long usize = buffer->size, i=0;
+
+ if (!buffer->pages)
+ return -EINVAL;
+
+ do {
+ ret = vm_insert_page(vma, uaddr, buffer->pages[i++]);
+ if (ret) {
+ DRM_ERROR("failed to remap user space.\n");
+ return ret;
+ }
+
+ uaddr += PAGE_SIZE;
+ usize -= PAGE_SIZE;
+ } while (usize > 0);
+ }
+ return 0;
+}
+
+static struct dma_buf_ops exynos_dmabuf_ops = {
+ .mmap = exynos_drm_gem_dmabuf_mmap,
+ .map_dma_buf = exynos_gem_map_dma_buf,
+ .unmap_dma_buf = exynos_gem_unmap_dma_buf,
+ .kmap = exynos_gem_dmabuf_kmap,
+ .kmap_atomic = exynos_gem_dmabuf_kmap_atomic,
+ .kunmap = exynos_gem_dmabuf_kunmap,
+ .kunmap_atomic = exynos_gem_dmabuf_kunmap_atomic,
+ .release = exynos_dmabuf_release,
+};
+
+struct dma_buf *exynos_dmabuf_prime_export(struct drm_device *drm_dev,
+ struct drm_gem_object *obj, int flags)
+{
+ struct exynos_drm_gem_obj *exynos_gem_obj = to_exynos_gem_obj(obj);
+
+ return dma_buf_export(exynos_gem_obj, &exynos_dmabuf_ops,
+ exynos_gem_obj->base.size, 0666);
+}
+
+struct drm_gem_object *exynos_dmabuf_prime_import(struct drm_device *drm_dev,
+ struct dma_buf *dma_buf)
+{
+ struct dma_buf_attachment *attach;
+ struct sg_table *sgt;
+ struct scatterlist *sgl;
+ struct exynos_drm_gem_obj *exynos_gem_obj;
+ struct exynos_drm_gem_buf *buffer;
+ struct page *page;
+ int ret, i = 0;
+
+ DRM_DEBUG_PRIME("%s\n", __FILE__);
+
+ /* is this one of own objects? */
+ if (dma_buf->ops == &exynos_dmabuf_ops) {
+ struct drm_gem_object *obj;
+
+ exynos_gem_obj = dma_buf->priv;
+ obj = &exynos_gem_obj->base;
+
+ /* is it from our device? */
+ if (obj->dev == drm_dev) {
+ drm_gem_object_reference(obj);
+ return obj;
+ }
+ }
+
+ attach = dma_buf_attach(dma_buf, drm_dev->dev);
+ if (IS_ERR(attach))
+ return ERR_PTR(-EINVAL);
+
+
+ sgt = dma_buf_map_attachment(attach, DMA_BIDIRECTIONAL);
+ if (IS_ERR(sgt)) {
+ ret = PTR_ERR(sgt);
+ goto err_buf_detach;
+ }
+
+ buffer = kzalloc(sizeof(*buffer), GFP_KERNEL);
+ if (!buffer) {
+ DRM_ERROR("failed to allocate exynos_drm_gem_buf.\n");
+ ret = -ENOMEM;
+ goto err_unmap_attach;
+ }
+
+ buffer->pages = kzalloc(sizeof(*page) * sgt->nents, GFP_KERNEL);
+ if (!buffer->pages) {
+ DRM_ERROR("failed to allocate pages.\n");
+ ret = -ENOMEM;
+ goto err_free_buffer;
+ }
+
+ exynos_gem_obj = exynos_drm_gem_init(drm_dev, dma_buf->size);
+ if (!exynos_gem_obj) {
+ ret = -ENOMEM;
+ goto err_free_pages;
+ }
+
+ sgl = sgt->sgl;
+ buffer->dma_addr = sg_dma_address(sgl);
+
+ while (i < sgt->nents) {
+ buffer->pages[i] = sg_page(sgl);
+ buffer->size += sg_dma_len(sgl);
+ sgl = sg_next(sgl);
+ i++;
+ }
+
+ exynos_gem_obj->buffer = buffer;
+ buffer->sgt = sgt;
+ exynos_gem_obj->base.import_attach = attach;
+
+ DRM_DEBUG_PRIME("dma_addr = 0x%x, size = 0x%lx\n", buffer->dma_addr,
+ buffer->size);
+
+ return &exynos_gem_obj->base;
+
+err_free_pages:
+ kfree(buffer->pages);
+ buffer->pages = NULL;
+err_free_buffer:
+ kfree(buffer);
+ buffer = NULL;
+err_unmap_attach:
+ dma_buf_unmap_attachment(attach, sgt, DMA_BIDIRECTIONAL);
+err_buf_detach:
+ dma_buf_detach(dma_buf, attach);
+ return ERR_PTR(ret);
+}
+
+MODULE_AUTHOR("Inki Dae <inki.dae@samsung.com>");
+MODULE_DESCRIPTION("Samsung SoC DRM DMABUF Module");
+MODULE_LICENSE("GPL");
--- /dev/null
+/* exynos_drm_dmabuf.h
+ *
+ * Copyright (c) 2012 Samsung Electronics Co., Ltd.
+ * Author: Inki Dae <inki.dae@samsung.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * VA LINUX SYSTEMS AND/OR ITS SUPPLIERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef _EXYNOS_DRM_DMABUF_H_
+#define _EXYNOS_DRM_DMABUF_H_
+
+#ifdef CONFIG_DRM_EXYNOS_DMABUF
+struct dma_buf *exynos_dmabuf_prime_export(struct drm_device *drm_dev,
+ struct drm_gem_object *obj, int flags);
+
+struct drm_gem_object *exynos_dmabuf_prime_import(struct drm_device *drm_dev,
+ struct dma_buf *dma_buf);
+#else
+#define exynos_dmabuf_prime_export NULL
+#define exynos_dmabuf_prime_import NULL
+#endif
+#endif
#include "exynos_drm_gem.h"
#include "exynos_drm_plane.h"
#include "exynos_drm_vidi.h"
+#include "exynos_drm_dmabuf.h"
#define DRIVER_NAME "exynos"
#define DRIVER_DESC "Samsung SoC DRM"
{
DRM_DEBUG_DRIVER("%s\n", __FILE__);
+ drm_prime_init_file_private(&file->prime);
+
return exynos_drm_subdrv_open(dev, file);
}
e->base.destroy(&e->base);
}
}
+ drm_prime_destroy_file_private(&file->prime);
spin_unlock_irqrestore(&dev->event_lock, flags);
exynos_drm_subdrv_close(dev, file);
static struct drm_driver exynos_drm_driver = {
.driver_features = DRIVER_HAVE_IRQ | DRIVER_BUS_PLATFORM |
- DRIVER_MODESET | DRIVER_GEM,
+ DRIVER_MODESET | DRIVER_GEM | DRIVER_PRIME,
.load = exynos_drm_load,
.unload = exynos_drm_unload,
.open = exynos_drm_open,
.dumb_create = exynos_drm_gem_dumb_create,
.dumb_map_offset = exynos_drm_gem_dumb_map_offset,
.dumb_destroy = exynos_drm_gem_dumb_destroy,
+ .prime_handle_to_fd = drm_gem_prime_handle_to_fd,
+ .prime_fd_to_handle = drm_gem_prime_fd_to_handle,
+ .gem_prime_export = exynos_dmabuf_prime_export,
+ .gem_prime_import = exynos_dmabuf_prime_import,
.ioctls = exynos_ioctls,
.fops = &exynos_drm_driver_fops,
.name = DRIVER_NAME,
.minor = DRIVER_MINOR,
};
+#ifdef CONFIG_EXYNOS_IOMMU
+static int iommu_init(struct platform_device *pdev)
+{
+ /* DRM device expects a IOMMU mapping to be already
+ * created in FIMD. Else this function should
+ * throw an error.
+ */
+ if (exynos_drm_common_mapping==NULL) {
+ printk(KERN_ERR "exynos drm common mapping is invalid\n");
+ return -1;
+ }
+
+ if (!s5p_create_iommu_mapping(&pdev->dev, 0,
+ 0, 0, exynos_drm_common_mapping)) {
+ printk(KERN_ERR "failed to create IOMMU mapping\n");
+ return -1;
+ }
+
+ return 0;
+}
+#endif
+
static int exynos_drm_platform_probe(struct platform_device *pdev)
{
DRM_DEBUG_DRIVER("%s\n", __FILE__);
+#ifdef CONFIG_EXYNOS_IOMMU
+ if (iommu_init(pdev)) {
+ DRM_ERROR("failed to initialize IOMMU\n");
+ return -ENODEV;
+ }
+#endif
+
exynos_drm_driver.num_ioctls = DRM_ARRAY_SIZE(exynos_ioctls);
return drm_platform_init(&exynos_drm_driver, pdev);
#include <linux/module.h>
#include "drm.h"
+#ifdef CONFIG_EXYNOS_IOMMU
+#include <mach/sysmmu.h>
+#include <linux/of_platform.h>
+#endif
#define MAX_CRTC 3
#define MAX_PLANE 5
extern struct platform_driver mixer_driver;
extern struct platform_driver exynos_drm_common_hdmi_driver;
extern struct platform_driver vidi_driver;
+#ifdef CONFIG_EXYNOS_IOMMU
+extern struct dma_iommu_mapping *exynos_drm_common_mapping;
+#endif
#endif
struct exynos_drm_gem_obj *exynos_gem_obj;
};
+static int
+exynos_drm_fb_mmap(struct fb_info *info, struct vm_area_struct * vma)
+{
+ int ret;
+
+ vma->vm_pgoff = 0;
+ ret = dma_mmap_writecombine(info->device, vma, info->screen_base,
+ info->fix.smem_start, vma->vm_end - vma->vm_start);
+ if (ret)
+ printk(KERN_ERR "Remapping memory failed, error: %d\n", ret);
+
+ return ret;
+}
+
static struct fb_ops exynos_drm_fb_ops = {
.owner = THIS_MODULE,
+ .fb_mmap = exynos_drm_fb_mmap,
.fb_fillrect = cfb_fillrect,
.fb_copyarea = cfb_copyarea,
.fb_imageblit = cfb_imageblit,
return 0;
}
+#ifdef CONFIG_EXYNOS_IOMMU
+static int iommu_init(struct platform_device *pdev)
+{
+ struct platform_device *pds;
+
+ pds = find_sysmmu_dt(pdev, "sysmmu");
+ if (pds==NULL) {
+ printk(KERN_ERR "No sysmmu found\n");
+ return -1;
+ }
+
+ platform_set_sysmmu(&pds->dev, &pdev->dev);
+ exynos_drm_common_mapping = s5p_create_iommu_mapping(&pdev->dev,
+ 0x20000000, SZ_128M, 4,
+ exynos_drm_common_mapping);
+
+ if (!exynos_drm_common_mapping) {
+ printk(KERN_ERR "IOMMU mapping not created\n");
+ return -1;
+ }
+
+ return 0;
+}
+#endif
static int __devinit fimd_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct exynos_drm_fimd_pdata *pdata;
struct exynos_drm_panel_info *panel;
struct resource *res;
+ struct clk *clk_parent;
int win;
int ret = -EINVAL;
+#ifdef CONFIG_EXYNOS_IOMMU
+ ret = iommu_init(pdev);
+ if (ret < 0) {
+ dev_err(dev, "failed to initialize IOMMU\n");
+ return -ENODEV;
+ }
+#endif
DRM_DEBUG_KMS("%s\n", __FILE__);
pdata = pdev->dev.platform_data;
goto err_bus_clk;
}
+ clk_parent = clk_get(NULL, "mout_mpll_user");
+ if (IS_ERR(clk_parent)) {
+ ret = PTR_ERR(clk_parent);
+ goto err_clk;
+ }
+
+ if (clk_set_parent(ctx->lcd_clk, clk_parent)) {
+ ret = PTR_ERR(ctx->lcd_clk);
+ goto err_clk;
+ }
+
+ if (clk_set_rate(ctx->lcd_clk, pdata->clock_rate)) {
+ ret = PTR_ERR(ctx->lcd_clk);
+ goto err_clk;
+ }
+
+ clk_put(clk_parent);
+
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
if (!res) {
dev_err(dev, "failed to find registers\n");
goto err_req_region_io;
}
- res = platform_get_resource(pdev, IORESOURCE_IRQ, 0);
+ res = platform_get_resource(pdev, IORESOURCE_IRQ, 1);
if (!res) {
dev_err(dev, "irq request failed.\n");
goto err_req_region_irq;
for (win = 0; win < WINDOWS_NR; win++)
fimd_clear_win(ctx, win);
+ if (pdata->panel_type == DP_LCD)
+ writel(DPCLKCON_ENABLE, ctx->regs + DPCLKCON);
+
exynos_drm_subdrv_register(subdrv);
return 0;
}
#endif
+static struct platform_device_id exynos_drm_driver_ids[] = {
+ {
+ .name = "exynos4-fb",
+ }, {
+ .name = "exynos5-fb",
+ },
+ {},
+};
+MODULE_DEVICE_TABLE(platform, exynos_drm_driver_ids);
+
static const struct dev_pm_ops fimd_pm_ops = {
SET_SYSTEM_SLEEP_PM_OPS(fimd_suspend, fimd_resume)
SET_RUNTIME_PM_OPS(fimd_runtime_suspend, fimd_runtime_resume, NULL)
struct platform_driver fimd_driver = {
.probe = fimd_probe,
.remove = __devexit_p(fimd_remove),
+ .id_table = exynos_drm_driver_ids,
.driver = {
- .name = "exynos4-fb",
+ .name = "exynos-drm-fimd",
.owner = THIS_MODULE,
.pm = &fimd_pm_ops,
},
return roundup(size, PAGE_SIZE);
}
-static struct page **exynos_gem_get_pages(struct drm_gem_object *obj,
+struct page **exynos_gem_get_pages(struct drm_gem_object *obj,
gfp_t gfpmask)
{
struct inode *inode;
}
npages = obj->size >> PAGE_SHIFT;
+ buf->page_size = PAGE_SIZE;
buf->sgt = kzalloc(sizeof(struct sg_table), GFP_KERNEL);
if (!buf->sgt) {
sgl = sg_next(sgl);
}
+ /* Map the SGT to create a IOMMU mapping for this buffer */
+ ret = dma_map_sg(obj->dev->dev, buf->sgt->sgl, buf->sgt->orig_nents, DMA_BIDIRECTIONAL);
+ if (!ret) {
+ DRM_ERROR("failed to map sg\n");
+ ret = -ENOMEM;
+ goto err1;
+ }
+ buf->dma_addr = buf->sgt->sgl->dma_address;
+
/* add some codes for UNCACHED type here. TODO */
buf->pages = pages;
struct exynos_drm_gem_obj *exynos_gem_obj = to_exynos_gem_obj(obj);
struct exynos_drm_gem_buf *buf = exynos_gem_obj->buffer;
+ /* Unmap the SGT to remove the IOMMU mapping created for this buffer */
+ dma_unmap_sg(obj->dev->dev, buf->sgt->sgl, buf->sgt->orig_nents, DMA_BIDIRECTIONAL);
+
/*
* if buffer typs is EXYNOS_BO_NONCONTIG then release all pages
* allocated at gem fault handler.
exynos_gem_obj = NULL;
}
-static struct exynos_drm_gem_obj *exynos_drm_gem_init(struct drm_device *dev,
+struct exynos_drm_gem_obj *exynos_drm_gem_init(struct drm_device *dev,
unsigned long size)
{
struct exynos_drm_gem_obj *exynos_gem_obj;
void exynos_drm_gem_free_object(struct drm_gem_object *obj)
{
+ struct exynos_drm_gem_obj *exynos_gem_obj;
+ struct exynos_drm_gem_buf *buf;
+
DRM_DEBUG_KMS("%s\n", __FILE__);
+ exynos_gem_obj = to_exynos_gem_obj(obj);
+ buf = exynos_gem_obj->buffer;
+
+ if (obj->import_attach)
+ drm_prime_gem_destroy(obj, buf->sgt);
+
exynos_drm_gem_destroy(to_exynos_gem_obj(obj));
}
* with DRM_IOCTL_MODE_CREATE_DUMB command.
*/
- args->pitch = args->width * args->bpp >> 3;
+ args->pitch = args->width * ALIGN(args->bpp, 8) >> 3;
+
args->size = PAGE_ALIGN(args->pitch * args->height);
exynos_gem_obj = exynos_drm_gem_create(dev, args->flags, args->size);
* device address with IOMMU.
* @sgt: sg table to transfer page data.
* @pages: contain all pages to allocated memory region.
+ * @page_size: could be 4K, 64K or 1MB.
* @size: size of allocated memory region.
*/
struct exynos_drm_gem_buf {
dma_addr_t dma_addr;
struct sg_table *sgt;
struct page **pages;
+ unsigned long page_size;
unsigned long size;
};
unsigned int flags;
};
+struct page **exynos_gem_get_pages(struct drm_gem_object *obj, gfp_t gfpmask);
+
/* destroy a buffer with gem object */
void exynos_drm_gem_destroy(struct exynos_drm_gem_obj *exynos_gem_obj);
+/* create a private gem object and initialize it. */
+struct exynos_drm_gem_obj *exynos_drm_gem_init(struct drm_device *dev,
+ unsigned long size);
+
/* create a new buffer with gem object */
struct exynos_drm_gem_obj *exynos_drm_gem_create(struct drm_device *dev,
unsigned int flags,
--- /dev/null
+menuconfig VITHAR
+ tristate "Enable Vithar DDK"
+ help
+ Choose this option to enable 3D rendering with vithar DDK.
+
+config VITHAR_DEVICE_NODE_CREATION_IN_RUNTIME
+ bool "Enable runtime device file creation by using UDEV"
+ depends on VITHAR
+ default y
+ help
+ Choose this option to create device file under dev folder in runtime. Must be yes for Android.
+
+config VITHAR_RT_PM
+ bool "Enable Runtime power management"
+ depends on VITHAR
+ help
+ Choose this option to enable runtime power management on vithar DDK.
+
+config VITHAR_DVFS
+ bool "Enable Dynamic frequency and volatge scaling"
+ depends on VITHAR
+ help
+ Choose this option to enable dynamic frequency and volatge scaling
--- /dev/null
+obj-$(CONFIG_VITHAR) += kbase/ osk/ uk/
--- /dev/null
+obj-y += src/
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Software workarounds configuration for Hardware issues.
+ */
+
+#ifndef _BASE_HWCONFIG_H_
+#define _BASE_HWCONFIG_H_
+
+#include <malisw/mali_malisw.h>
+
+/**
+ * List of all workarounds.
+ *
+ */
+
+typedef enum base_hw_issue {
+
+ /* Tiler triggers a fault if the scissor rectangle is empty. */
+ BASE_HW_ISSUE_5699,
+
+ /* The current version of the model doesn't support Soft-Stop */
+ BASE_HW_ISSUE_5736,
+
+ /* Need way to guarantee that all previously-translated memory accesses are commited */
+ BASE_HW_ISSUE_6367,
+
+ /* Unaligned load stores crossing 128 bit boundaries will fail */
+ BASE_HW_ISSUE_6402,
+
+ /* On job complete with non-done the cache is not flushed */
+ BASE_HW_ISSUE_6787,
+
+ /* The clamp integer coordinate flag bit of the sampler descriptor is reserved */
+ BASE_HW_ISSUE_7144,
+
+ /* Write of PRFCNT_CONFIG_MODE_MANUAL to PRFCNT_CONFIG causes a instrumentation dump if
+ PRFCNT_TILER_EN is enabled */
+ BASE_HW_ISSUE_8186,
+
+ /* TIB: Reports faults from a vtile which has not yet been allocated */
+ BASE_HW_ISSUE_8245,
+
+ /* WLMA memory goes wrong when run on shader cores other than core 0. */
+ BASE_HW_ISSUE_8250,
+
+ /* Hierz doesn't work when stenciling is enabled */
+ BASE_HW_ISSUE_8260,
+
+ /* Livelock in L0 icache */
+ BASE_HW_ISSUE_8280,
+
+ /* uTLB deadlock could occur when writing to an invalid page at the same time as
+ * access to a valid page in the same uTLB cache line ( == 4 PTEs == 16K block of mapping) */
+ BASE_HW_ISSUE_8316,
+
+ /* HT: TERMINATE for RUN command ignored if previous LOAD_DESCRIPTOR is still executing */
+ BASE_HW_ISSUE_8394,
+
+ /* CSE : Sends a TERMINATED response for a task that should not be terminated */
+ /* (Note that PRLAM-8379 also uses this workaround) */
+ BASE_HW_ISSUE_8401,
+
+ /* Repeatedly Soft-stopping a job chain consisting of (Vertex Shader, Cache Flush, Tiler)
+ * jobs causes 0x58 error on tiler job. */
+ BASE_HW_ISSUE_8408,
+
+ /* Disable the Pause Buffer in the LS pipe. */
+ BASE_HW_ISSUE_8443,
+
+ /* Stencil test enable 1->0 sticks */
+ BASE_HW_ISSUE_8456,
+
+ /* Tiler heap issue using FBOs or multiple processes using the tiler simultaneously */
+ BASE_HW_ISSUE_8564,
+
+ /* Livelock issue using atomic instructions (particularly when using atomic_cmpxchg as a spinlock) */
+ BASE_HW_ISSUE_8791,
+
+ /* Fused jobs are not supported (for various reasons) */
+ /* Jobs with relaxed dependencies do not support soft-stop */
+ /* (Note that PRLAM-8803, PRLAM-8393, PRLAM-8559, PRLAM-8601 & PRLAM-8607 all use this work-around) */
+ BASE_HW_ISSUE_8803,
+
+ /* Blend shader output is wrong for certain formats */
+ BASE_HW_ISSUE_8833,
+
+ /* Occlusion queries can create false 0 result in boolean and counter modes */
+ BASE_HW_ISSUE_8879,
+
+ /* Output has half intensity with blend shaders enabled on 8xMSAA. */
+ BASE_HW_ISSUE_8896,
+
+ /* 8xMSAA does not work with CRC */
+ BASE_HW_ISSUE_8975,
+
+ /* Boolean occlusion queries don't work properly due to sdc issue. */
+ BASE_HW_ISSUE_8986,
+
+ /* Change in RMUs in use causes problems related with the core's SDC */
+ BASE_HW_ISSUE_8987,
+
+ /* Occlusion query result is not updated if color writes are disabled. */
+ BASE_HW_ISSUE_9010,
+
+ /* Problem with number of work registers in the RSD if set to 0 */
+ BASE_HW_ISSUE_9275,
+
+ /* Compute endpoint has a 4-deep queue of tasks, meaning a soft stop won't complete until all 4 tasks have completed */
+ BASE_HW_ISSUE_9435,
+
+ /* HT: Tiler returns TERMINATED for command that hasn't been terminated */
+ BASE_HW_ISSUE_9510,
+
+ /* Occasionally the GPU will issue multiple page faults for the same address before the MMU page table has been read by the GPU */
+ BASE_HW_ISSUE_9630,
+
+ /* The BASE_HW_ISSUE_END value must be the last issue listed in this enumeration
+ * and must be the last value in each array that contains the list of workarounds
+ * for a particular HW version.
+ */
+ BASE_HW_ISSUE_END
+} base_hw_issue;
+
+
+/**
+ * Workarounds configuration for each HW revision
+ */
+
+/* Mali T60x/T65x r0p0-15dev0 - 2011-W39-stable-9 */
+static const base_hw_issue base_hw_issues_t60x_t65x_r0p0_15dev0[] =
+{
+ BASE_HW_ISSUE_5699,
+ BASE_HW_ISSUE_6367,
+ BASE_HW_ISSUE_6402,
+ BASE_HW_ISSUE_6787,
+ BASE_HW_ISSUE_7144,
+ BASE_HW_ISSUE_8186,
+ BASE_HW_ISSUE_8245,
+ BASE_HW_ISSUE_8250,
+ BASE_HW_ISSUE_8260,
+ BASE_HW_ISSUE_8280,
+ BASE_HW_ISSUE_8316,
+ BASE_HW_ISSUE_8394,
+ BASE_HW_ISSUE_8401,
+ BASE_HW_ISSUE_8408,
+ BASE_HW_ISSUE_8443,
+ BASE_HW_ISSUE_8456,
+ BASE_HW_ISSUE_8564,
+ BASE_HW_ISSUE_8791,
+ BASE_HW_ISSUE_8803,
+ BASE_HW_ISSUE_8833,
+ BASE_HW_ISSUE_8896,
+ BASE_HW_ISSUE_8975,
+ BASE_HW_ISSUE_8986,
+ BASE_HW_ISSUE_8987,
+ BASE_HW_ISSUE_9010,
+ BASE_HW_ISSUE_9275,
+ BASE_HW_ISSUE_9435,
+ BASE_HW_ISSUE_9510,
+ BASE_HW_ISSUE_9630,
+ /* List of hardware issues must end with BASE_HW_ISSUE_END */
+ BASE_HW_ISSUE_END
+};
+
+/* Mali T60x/T65x r0p0-00rel0 - 2011-W46-stable-13c */
+static const base_hw_issue base_hw_issues_t60x_t65x_r0p0_eac[] =
+{
+ BASE_HW_ISSUE_5699,
+ BASE_HW_ISSUE_6367,
+ BASE_HW_ISSUE_6402,
+ BASE_HW_ISSUE_6787,
+ BASE_HW_ISSUE_8186,
+ BASE_HW_ISSUE_8245,
+ BASE_HW_ISSUE_8260,
+ BASE_HW_ISSUE_8280,
+ BASE_HW_ISSUE_8316,
+ BASE_HW_ISSUE_8564,
+ BASE_HW_ISSUE_8803,
+ BASE_HW_ISSUE_9010,
+ BASE_HW_ISSUE_9275,
+ BASE_HW_ISSUE_9435,
+ BASE_HW_ISSUE_9510,
+ /* List of hardware issues must end with BASE_HW_ISSUE_END */
+ BASE_HW_ISSUE_END
+};
+
+/* Mali T65x r0p1 */
+static const base_hw_issue base_hw_issues_t65x_r0p1[] =
+{
+ BASE_HW_ISSUE_5699,
+ BASE_HW_ISSUE_6367,
+ BASE_HW_ISSUE_6402,
+ BASE_HW_ISSUE_6787,
+ BASE_HW_ISSUE_8186,
+ BASE_HW_ISSUE_8245,
+ BASE_HW_ISSUE_8260,
+ BASE_HW_ISSUE_8280,
+ BASE_HW_ISSUE_8316,
+ BASE_HW_ISSUE_8564,
+ BASE_HW_ISSUE_8803,
+ BASE_HW_ISSUE_9010,
+ BASE_HW_ISSUE_9275,
+ BASE_HW_ISSUE_9435,
+ BASE_HW_ISSUE_9510,
+ /* List of hardware issues must end with BASE_HW_ISSUE_END */
+ BASE_HW_ISSUE_END
+};
+
+/* Mali T60x/T65x r1p0-00rel0 */
+static const base_hw_issue base_hw_issues_t60x_t65x_r1p0[] =
+{
+ BASE_HW_ISSUE_6367,
+ BASE_HW_ISSUE_6402,
+ BASE_HW_ISSUE_8803,
+ BASE_HW_ISSUE_9435,
+ BASE_HW_ISSUE_9510,
+ /* List of hardware issues must end with BASE_HW_ISSUE_END */
+ BASE_HW_ISSUE_END
+};
+
+/* Mali T62x r0p0 */
+static const base_hw_issue base_hw_issues_t62x_r0p0[] =
+{
+ BASE_HW_ISSUE_6367,
+ BASE_HW_ISSUE_6402,
+ BASE_HW_ISSUE_8803,
+ BASE_HW_ISSUE_9435,
+ BASE_HW_ISSUE_9510,
+ /* List of hardware issues must end with BASE_HW_ISSUE_END */
+ BASE_HW_ISSUE_END
+};
+
+/* Mali T67x r0p0 */
+static const base_hw_issue base_hw_issues_t67x_r0p0[] =
+{
+ BASE_HW_ISSUE_6367,
+ BASE_HW_ISSUE_6402,
+ BASE_HW_ISSUE_8803,
+ BASE_HW_ISSUE_9435,
+ BASE_HW_ISSUE_9510,
+ /* List of hardware issues must end with BASE_HW_ISSUE_END */
+ BASE_HW_ISSUE_END
+};
+
+#if !MALI_BACKEND_KERNEL
+
+/* Model configuration
+ *
+ * note: We can only know that the model is used at compile-time
+ */
+
+static const base_hw_issue base_hw_issues_model[] =
+{
+ BASE_HW_ISSUE_5736,
+ BASE_HW_ISSUE_8260,
+ BASE_HW_ISSUE_8316,
+ BASE_HW_ISSUE_8394,
+ BASE_HW_ISSUE_8803,
+ /* NOTE: Model is fixed for BASE_HW_ISSUE_8975, but EGL is currently broken, see MIDEGL-868 */
+ BASE_HW_ISSUE_8975,
+ BASE_HW_ISSUE_8987,
+ BASE_HW_ISSUE_9010,
+ BASE_HW_ISSUE_9275,
+ BASE_HW_ISSUE_9435,
+ BASE_HW_ISSUE_9510, /* TODO: Review - should be disabled for model? */
+ /* List of hardware issues must end with BASE_HW_ISSUE_END */
+ BASE_HW_ISSUE_END
+};
+
+#endif /* !MALI_BACKEND_KERNEL */
+
+#endif /* _BASE_HWCONFIG_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Base structures shared with the kernel.
+ */
+
+#ifndef _BASE_KERNEL_H_
+#define _BASE_KERNEL_H_
+
+#include <kbase/src/mali_base_mem_priv.h>
+
+/*
+ * Dependency stuff, keep it private for now. May want to expose it if
+ * we decide to make the number of semaphores a configurable
+ * option.
+ */
+#define BASEP_JD_SEM_PER_WORD_LOG2 5
+#define BASEP_JD_SEM_PER_WORD (1 << BASEP_JD_SEM_PER_WORD_LOG2)
+#define BASEP_JD_SEM_WORD_NR(x) ((x) >> BASEP_JD_SEM_PER_WORD_LOG2)
+#define BASEP_JD_SEM_MASK_IN_WORD(x) (1 << ((x) & (BASEP_JD_SEM_PER_WORD - 1)))
+#define BASEP_JD_SEM_ARRAY_SIZE BASEP_JD_SEM_WORD_NR(256)
+
+/* Size of the ring buffer */
+#define BASEP_JCTX_RB_NRPAGES 16
+
+#define BASE_GPU_NUM_TEXTURE_FEATURES_REGISTERS 3
+
+#define BASE_MAX_COHERENT_GROUPS 16
+
+#if defined CDBG_ASSERT
+#define LOCAL_ASSERT CDBG_ASSERT
+#elif defined OSK_ASSERT
+#define LOCAL_ASSERT OSK_ASSERT
+#else
+#error assert macro not defined!
+#endif
+
+#if defined OSK_PAGE_MASK
+ #define LOCAL_PAGE_LSB ~OSK_PAGE_MASK
+#else
+ #include <osu/mali_osu.h>
+
+ #if defined CONFIG_CPU_PAGE_SIZE_LOG2
+ #define LOCAL_PAGE_LSB ((1ul << CONFIG_CPU_PAGE_SIZE_LOG2) - 1)
+ #else
+ #error Failed to find page size
+ #endif
+#endif
+
+
+/**
+ * @addtogroup base_user_api User-side Base APIs
+ * @{
+ */
+
+/**
+ * @addtogroup base_user_api_memory User-side Base Memory APIs
+ * @{
+ */
+
+/**
+ * @brief Memory allocation, access/hint flags
+ *
+ * A combination of MEM_PROT/MEM_HINT flags must be passed to each allocator
+ * in order to determine the best cache policy. Some combinations are
+ * of course invalid (eg @c MEM_PROT_CPU_WR | @c MEM_HINT_CPU_RD),
+ * which defines a @a write-only region on the CPU side, which is
+ * heavily read by the CPU...
+ * Other flags are only meaningful to a particular allocator.
+ * More flags can be added to this list, as long as they don't clash
+ * (see ::BASE_MEM_FLAGS_NR_BITS for the number of the first free bit).
+ */
+typedef u32 base_mem_alloc_flags;
+
+
+/**
+ * @brief Memory allocation, access/hint flags
+ *
+ * See ::base_mem_alloc_flags.
+ *
+ */
+enum
+{
+ BASE_MEM_PROT_CPU_RD = (1U << 0), /**< Read access CPU side */
+ BASE_MEM_PROT_CPU_WR = (1U << 1), /**< Write access CPU side */
+ BASE_MEM_PROT_GPU_RD = (1U << 2), /**< Read access GPU side */
+ BASE_MEM_PROT_GPU_WR = (1U << 3), /**< Write access GPU side */
+ BASE_MEM_PROT_GPU_EX = (1U << 4), /**< Execute allowed on the GPU side */
+
+ BASE_MEM_HINT_CPU_RD = (1U << 5), /**< Heavily read CPU side */
+ BASE_MEM_HINT_CPU_WR = (1U << 6), /**< Heavily written CPU side */
+ BASE_MEM_HINT_GPU_RD = (1U << 7), /**< Heavily read GPU side */
+ BASE_MEM_HINT_GPU_WR = (1U << 8), /**< Heavily written GPU side */
+
+ BASEP_MEM_GROWABLE = (1U << 9), /**< Growable memory. This is a private flag that is set automatically. Not valid for PMEM. */
+ BASE_MEM_GROW_ON_GPF = (1U << 10), /**< Grow backing store on GPU Page Fault */
+
+ BASE_MEM_COHERENT_SYSTEM = (1U << 11),/**< Page coherence Outer shareable */
+ BASE_MEM_COHERENT_LOCAL = (1U << 12) /**< Page coherence Inner shareable */
+};
+
+/**
+ * @brief Memory types supported by @a base_tmem_import
+ *
+ * Each type defines what the supported handle type is.
+ *
+ * If any new type is added here ARM must be contacted
+ * to allocate a numeric value for it.
+ * Do not just add a new type without synchronizing with ARM
+ * as future releases from ARM might include other new types
+ * which could clash with your custom types.
+ */
+typedef enum base_tmem_import_type
+{
+ BASE_TMEM_IMPORT_TYPE_INVALID = 0,
+ /** UMP import. Handle type is ump_secure_id. */
+ BASE_TMEM_IMPORT_TYPE_UMP = 1,
+ /** UMM import. Handle type is a file descriptor (int) */
+ BASE_TMEM_IMPORT_TYPE_UMM = 2
+} base_tmem_import_type;
+
+/**
+ * Bits we can tag into a memory handle.
+ * We use the lower 12 bits as our handles are page-multiples, thus not using the 12 LSBs
+ */
+enum
+{
+ BASE_MEM_TAGS_MASK = ((1U << 12) - 1), /**< Mask to get hold of the tag bits/see if there are tag bits */
+ BASE_MEM_TAG_IMPORTED = (1U << 0) /**< Tagged as imported */
+ /* max 1u << 11 supported */
+};
+
+
+/**
+ * @brief Number of bits used as flags for base memory management
+ *
+ * Must be kept in sync with the ::base_mem_alloc_flags flags
+ */
+#define BASE_MEM_FLAGS_NR_BITS 13
+
+/**
+ * @brief Result codes of changing the size of the backing store allocated to a tmem region
+ */
+typedef enum base_backing_threshold_status
+{
+ BASE_BACKING_THRESHOLD_OK = 0, /**< Resize successful */
+ BASE_BACKING_THRESHOLD_ERROR_NOT_GROWABLE = -1, /**< Not a growable tmem object */
+ BASE_BACKING_THRESHOLD_ERROR_OOM = -2, /**< Increase failed due to an out-of-memory condition */
+ BASE_BACKING_THRESHOLD_ERROR_MAPPED = -3, /**< Resize attempted on buffer while it was mapped, which is not permitted */
+ BASE_BACKING_THRESHOLD_ERROR_INVALID_ARGUMENTS = -4 /**< Invalid arguments (not tmem, illegal size request, etc.) */
+} base_backing_threshold_status;
+
+/**
+ * @addtogroup base_user_api_memory_defered User-side Base Defered Memory Coherency APIs
+ * @{
+ */
+
+/**
+ * @brief a basic memory operation (sync-set).
+ *
+ * The content of this structure is private, and should only be used
+ * by the accessors.
+ */
+typedef struct base_syncset
+{
+ basep_syncset basep_sset;
+} base_syncset;
+
+/** @} end group base_user_api_memory_defered */
+
+/**
+ * Handle to represent imported memory object.
+ * Simple opague handle to imported memory, can't be used
+ * with anything but base_external_resource_init to bind to an atom.
+ */
+typedef struct base_import_handle
+{
+ struct
+ {
+ mali_addr64 handle;
+ } basep;
+} base_import_handle;
+
+
+/** @} end group base_user_api_memory */
+
+/**
+ * @addtogroup base_user_api_job_dispatch User-side Base Job Dispatcher APIs
+ * @{
+ */
+
+/**
+ * @brief A pre- or post- dual dependency.
+ *
+ * This structure is used to express either
+ * @li a single or dual pre-dependency (a job depending on one or two
+ * other jobs),
+ * @li a single or dual post-dependency (a job resolving a dependency
+ * for one or two other jobs).
+ *
+ * The dependency itself is specified as a u8, where 0 indicates no
+ * dependency. A single dependency is expressed by having one of the
+ * dependencies set to 0.
+ */
+typedef struct base_jd_dep {
+ u8 dep[2]; /**< pre/post dependencies */
+} base_jd_dep;
+
+/**
+ * @brief Per-job data
+ *
+ * This structure is used to store per-job data, and is completly unused
+ * by the Base driver. It can be used to store things such as callback
+ * function pointer, data to handle job completion. It is guaranteed to be
+ * untouched by the Base driver.
+ */
+typedef struct base_jd_udata
+{
+ u64 blob[2]; /**< per-job data array */
+} base_jd_udata;
+
+/**
+ * @brief Job chain hardware requirements.
+ *
+ * A job chain must specify what GPU features it needs to allow the
+ * driver to schedule the job correctly. By not specifying the
+ * correct settings can/will cause an early job termination. Multiple
+ * values can be ORed together to specify multiple requirements.
+ * Special case is ::BASE_JD_REQ_DEP, which is used to express complex
+ * dependencies, and that doesn't execute anything on the hardware.
+ */
+typedef u16 base_jd_core_req;
+
+/* Requirements that come from the HW */
+#define BASE_JD_REQ_DEP 0 /**< No requirement, dependency only */
+#define BASE_JD_REQ_FS (1U << 0) /**< Requires fragment shaders */
+/**
+ * Requires compute shaders
+ * This covers any of the following Midgard Job types:
+ * - Vertex Shader Job
+ * - Geometry Shader Job
+ * - An actual Compute Shader Job
+ *
+ * Compare this with @ref BASE_JD_REQ_ONLY_COMPUTE, which specifies that the
+ * job is specifically just the "Compute Shader" job type, and not the "Vertex
+ * Shader" nor the "Geometry Shader" job type.
+ */
+#define BASE_JD_REQ_CS (1U << 1)
+#define BASE_JD_REQ_T (1U << 2) /**< Requires tiling */
+#define BASE_JD_REQ_CF (1U << 3) /**< Requires cache flushes */
+#define BASE_JD_REQ_V (1U << 4) /**< Requires value writeback */
+
+/* SW-only requirements - the HW does not expose these as part of the job slot capabilities */
+/**
+ * SW Only requirement: this job chain might not be soft-stoppable (Non-Soft
+ * Stoppable), and so must be scheduled separately from all other job-chains
+ * that are soft-stoppable.
+ *
+ * In absence of this requirement, then the job-chain is assumed to be
+ * soft-stoppable. That is, if it does not release the GPU "soon after" it is
+ * soft-stopped, then it will be killed. In contrast, NSS job chains can
+ * release the GPU "a long time after" they are soft-stopped.
+ *
+ * "soon after" and "a long time after" are implementation defined, and
+ * configurable in the device driver by the system integrator.
+ */
+#define BASE_JD_REQ_NSS (1U << 5)
+
+/**
+ * SW Only requirement: the job chain requires a coherent core group. We don't
+ * mind which coherent core group is used.
+ */
+#define BASE_JD_REQ_COHERENT_GROUP (1U << 6)
+
+/**
+ * SW Only requirement: The performance counters should be enabled only when
+ * they are needed, to reduce power consumption.
+ */
+
+#define BASE_JD_REQ_PERMON (1U << 7)
+
+/**
+ * SW Only requirement: External resources are referenced by this atom.
+ * When external resources are referenced no syncsets can be bundled with the atom
+ * but should instead be part of a NULL jobs inserted into the dependency tree.
+ * The first pre_dep object must be configured for the external resouces to use,
+ * the second pre_dep object can be used to create other dependencies.
+ */
+#define BASE_JD_REQ_EXTERNAL_RESOURCES (1U << 8)
+
+/**
+ * SW Only requirement: Software defined job. Jobs with this bit set will not be submitted
+ * to the hardware but will cause some action to happen within the driver
+ */
+#define BASE_JD_REQ_SOFT_JOB (1U << 9)
+
+#define BASE_JD_REQ_SOFT_DUMP_CPU_GPU_TIME (BASE_JD_REQ_SOFT_JOB | 0x1)
+
+/**
+ * HW Requirement: Requires Compute shaders (but not Vertex or Geometry Shaders)
+ *
+ * This indicates that the Job Chain contains Midgard Jobs of the 'Compute Shaders' type.
+ *
+ * In contrast to @ref BASE_JD_REQ_CS, this does \b not indicate that the Job
+ * Chain contains 'Geometry Shader' or 'Vertex Shader' jobs.
+ *
+ * @note This is a more flexible variant of the @ref BASE_CONTEXT_HINT_ONLY_COMPUTE flag,
+ * allowing specific jobs to be marked as 'Only Compute' instead of the entire context
+ */
+#define BASE_JD_REQ_ONLY_COMPUTE (1U << 10)
+
+/**
+ * HW Requirement: Use the base_jd_atom::device_nr field to specify a
+ * particular core group
+ *
+ * If both BASE_JD_REQ_COHERENT_GROUP and this flag are set, this flag takes priority
+ *
+ * This is only guaranteed to work for BASE_JD_REQ_ONLY_COMPUTE atoms.
+ */
+#define BASE_JD_REQ_SPECIFIC_COHERENT_GROUP ( 1U << 11 )
+
+/**
+* These requirement bits are currently unused in base_jd_core_req (currently a u16)
+*/
+
+#define BASEP_JD_REQ_RESERVED_BIT12 ( 1U << 12 )
+#define BASEP_JD_REQ_RESERVED_BIT13 ( 1U << 13 )
+#define BASEP_JD_REQ_RESERVED_BIT14 ( 1U << 14 )
+#define BASEP_JD_REQ_RESERVED_BIT15 ( 1U << 15 )
+
+/**
+* Mask of all the currently unused requirement bits in base_jd_core_req.
+*/
+
+#define BASEP_JD_REQ_RESERVED ( BASEP_JD_REQ_RESERVED_BIT12 | BASEP_JD_REQ_RESERVED_BIT13 |\
+ BASEP_JD_REQ_RESERVED_BIT14 | BASEP_JD_REQ_RESERVED_BIT15 )
+
+
+/**
+ * @brief A single job chain, with pre/post dependendencies and mem ops
+ *
+ * This structure is used to describe a single job-chain to be submitted
+ * as part of a bag.
+ * It contains all the necessary information for Base to take care of this
+ * job-chain, including core requirements, priority, syncsets and
+ * dependencies.
+ */
+typedef struct base_jd_atom
+{
+ mali_addr64 jc; /**< job-chain GPU address */
+ base_jd_udata udata; /**< user data */
+ base_jd_dep pre_dep; /**< pre-dependencies */
+ base_jd_dep post_dep; /**< post-dependencies */
+ base_jd_core_req core_req; /**< core requirements */
+ u16 nr_syncsets; /**< nr of syncsets following the atom */
+ u16 nr_extres; /**< nr of external resources following the atom */
+
+ /** @brief Relative priority.
+ *
+ * A positive value requests a lower priority, whilst a negative value
+ * requests a higher priority. Only privileged processes may request a
+ * higher priority. For unprivileged processes, a negative priority will
+ * be interpreted as zero.
+ */
+ s8 prio;
+
+ /**
+ * @brief Device number to use, depending on @ref base_jd_core_req flags set.
+ *
+ * When BASE_JD_REQ_SPECIFIC_COHERENT_GROUP is set, a 'device' is one of
+ * the coherent core groups, and so this targets a particular coherent
+ * core-group. They are numbered from 0 to (mali_base_gpu_coherent_group_info::num_groups - 1),
+ * and the cores targeted by this device_nr will usually be those specified by
+ * (mali_base_gpu_coherent_group_info::group[device_nr].core_mask).
+ * Further, two atoms from different processes using the same \a device_nr
+ * at the same time will always target the same coherent core-group.
+ *
+ * There are exceptions to when the device_nr is ignored:
+ * - when any process in the system uses a BASE_JD_REQ_CS or
+ * BASE_JD_REQ_ONLY_COMPUTE atom that can run on all cores across all
+ * coherency groups (i.e. also does \b not have the
+ * BASE_JD_REQ_COHERENT_GROUP or BASE_JD_REQ_SPECIFIC_COHERENT_GROUP flags
+ * set). In this case, such atoms would block device_nr==1 being used due
+ * to restrictions on affinity, perhaps indefinitely. To ensure progress is
+ * made, the atoms targeted for device_nr 1 will instead be redirected to
+ * device_nr 0
+ * - When any process in the system is using 'NSS' (BASE_JD_REQ_NSS) atoms,
+ * because there'd be very high latency on atoms targeting a coregroup
+ * that is also in use by NSS atoms. To ensure progress is
+ * made, the atoms targeted for device_nr 1 will instead be redirected to
+ * device_nr 0
+ * - During certain HW workarounds, such as BASE_HW_ISSUE_8987, where
+ * BASE_JD_REQ_ONLY_COMPUTE atoms must not use the same cores as other
+ * atoms. In this case, all atoms are targeted to device_nr == min( num_groups, 1 )
+ *
+ * Note that the 'device' number for a coherent coregroup cannot exceed
+ * (BASE_MAX_COHERENT_GROUPS - 1).
+ */
+ u8 device_nr;
+} base_jd_atom;
+
+/* Structure definition works around the fact that C89 doesn't allow arrays of size 0 */
+typedef struct basep_jd_atom_ss
+{
+ base_jd_atom atom;
+ base_syncset syncsets[1];
+} basep_jd_atom_ss;
+
+typedef enum base_external_resource_access
+{
+ BASE_EXT_RES_ACCESS_SHARED,
+ BASE_EXT_RES_ACCESS_EXCLUSIVE
+} base_external_resource_access;
+
+typedef struct base_external_resource
+{
+ u64 ext_resource;
+} base_external_resource;
+
+/* Structure definition works around the fact that C89 doesn't allow arrays of size 0 */
+typedef struct basep_jd_atom_ext_res
+{
+ base_jd_atom atom;
+ base_external_resource resources[1];
+} basep_jd_atom_ext_res;
+
+static INLINE size_t base_jd_atom_size_ex(u32 syncset_count, u32 external_res_count)
+{
+ LOCAL_ASSERT( 0 == syncset_count || 0 == external_res_count );
+
+ return syncset_count ? offsetof(basep_jd_atom_ss, syncsets[0]) + (sizeof(base_syncset) * syncset_count) :
+ external_res_count ? offsetof(basep_jd_atom_ext_res, resources[0]) + (sizeof(base_external_resource) * external_res_count) :
+ sizeof(base_jd_atom);
+}
+
+/**
+ * @brief Atom size evaluator
+ *
+ * This function returns the size in bytes of a ::base_jd_atom
+ * containing @a n syncsets. It must be used to compute the size of a
+ * bag before allocation.
+ *
+ * @param nr the number of syncsets for this atom
+ * @return the atom size in bytes
+ */
+static INLINE size_t base_jd_atom_size(u32 nr)
+{
+ return base_jd_atom_size_ex(nr, 0);
+}
+
+/**
+ * @brief Atom syncset accessor
+ *
+ * This function returns a pointer to the nth syncset allocated
+ * together with an atom.
+ *
+ * @param[in] atom The allocated atom
+ * @param n The number of the syncset to be returned
+ * @return a pointer to the nth syncset.
+ */
+static INLINE base_syncset *base_jd_get_atom_syncset(base_jd_atom *atom, int n)
+{
+ LOCAL_ASSERT(atom != NULL);
+ LOCAL_ASSERT(0 == (atom->core_req & BASE_JD_REQ_EXTERNAL_RESOURCES));
+ LOCAL_ASSERT( (n >= 0) && (n <= atom->nr_syncsets) );
+ return &((basep_jd_atom_ss *)atom)->syncsets[n];
+}
+
+/**
+ * @brief Atom external resource accessor
+ *
+ * This functions returns a pointer to the nth external resource tracked by the atom.
+ *
+ * @param[in] atom The allocated atom
+ * @param n The number of the external resource to return a pointer to
+ * @return a pointer to the nth external resource
+ */
+static INLINE base_external_resource *base_jd_get_external_resource(base_jd_atom *atom, int n)
+{
+ LOCAL_ASSERT(atom != NULL);
+ LOCAL_ASSERT(BASE_JD_REQ_EXTERNAL_RESOURCES == (atom->core_req & BASE_JD_REQ_EXTERNAL_RESOURCES));
+ LOCAL_ASSERT( (n >= 0) && (n <= atom->nr_extres) );
+ return &((basep_jd_atom_ext_res*)atom)->resources[n];
+}
+
+/**
+ * @brief External resource info initialization.
+ *
+ * Sets up a external resource object to reference
+ * a memory allocation and the type of access requested.
+ *
+ * @param[in] res The resource object to initialize
+ * @param address The GPU VA of the external memory
+ * @param access The type of access requested
+ */
+static INLINE void base_external_resource_init(base_external_resource * res, base_import_handle handle, base_external_resource_access access)
+{
+ mali_addr64 address;
+ address = handle.basep.handle;
+
+ LOCAL_ASSERT(res != NULL);
+ LOCAL_ASSERT(0 == (address & LOCAL_PAGE_LSB));
+ LOCAL_ASSERT(access == BASE_EXT_RES_ACCESS_SHARED || access == BASE_EXT_RES_ACCESS_EXCLUSIVE);
+
+ res->ext_resource = address | (access & LOCAL_PAGE_LSB);
+}
+
+/**
+ * @brief Next atom accessor
+ *
+ * This function returns a pointer to the next allocated atom. It
+ * relies on the fact that the current atom has been correctly
+ * initialized (relies on the base_jd_atom::nr_syncsets field).
+ *
+ * @param[in] atom The allocated atom
+ * @return a pointer to the next atom.
+ */
+static INLINE base_jd_atom *base_jd_get_next_atom(base_jd_atom *atom)
+{
+ LOCAL_ASSERT(atom != NULL);
+ return (atom->core_req & BASE_JD_REQ_EXTERNAL_RESOURCES) ? (base_jd_atom *)base_jd_get_external_resource(atom, atom->nr_extres) :
+ (base_jd_atom *)base_jd_get_atom_syncset(atom, atom->nr_syncsets);
+}
+
+/**
+ * @brief Job chain event code bits
+ * Defines the bits used to create ::base_jd_event_code
+ */
+enum
+{
+ BASE_JD_SW_EVENT_KERNEL = (1u << 15), /**< Kernel side event */
+ BASE_JD_SW_EVENT = (1u << 14), /**< SW defined event */
+ BASE_JD_SW_EVENT_SUCCESS = (1u << 13), /**< Event idicates success (SW events only) */
+ BASE_JD_SW_EVENT_JOB = (0u << 11), /**< Job related event */
+ BASE_JD_SW_EVENT_BAG = (1u << 11), /**< Bag related event */
+ BASE_JD_SW_EVENT_INFO = (2u << 11), /**< Misc/info event */
+ BASE_JD_SW_EVENT_RESERVED = (3u << 11), /**< Reserved event type */
+ BASE_JD_SW_EVENT_TYPE_MASK = (3u << 11) /**< Mask to extract the type from an event code */
+};
+
+/**
+ * @brief Job chain event codes
+ *
+ * HW and low-level SW events are represented by event codes.
+ * The status of jobs which succeeded are also represented by
+ * an event code (see ::BASE_JD_EVENT_DONE).
+ * Events are usually reported as part of a ::base_jd_event.
+ *
+ * The event codes are encoded in the following way:
+ * @li 10:0 - subtype
+ * @li 12:11 - type
+ * @li 13 - SW success (only valid if the SW bit is set)
+ * @li 14 - SW event (HW event if not set)
+ * @li 15 - Kernel event (should never be seen in userspace)
+ *
+ * Events are split up into ranges as follows:
+ * - BASE_JD_EVENT_RANGE_\<description\>_START
+ * - BASE_JD_EVENT_RANGE_\<description\>_END
+ *
+ * \a code is in \<description\>'s range when:
+ * - <tt>BASE_JD_EVENT_RANGE_\<description\>_START <= code < BASE_JD_EVENT_RANGE_\<description\>_END </tt>
+ *
+ * Ranges can be asserted for adjacency by testing that the END of the previous
+ * is equal to the START of the next. This is useful for optimizing some tests
+ * for range.
+ *
+ * A limitation is that the last member of this enum must explicitly be handled
+ * (with an assert-unreachable statement) in switch statements that use
+ * variables of this type. Otherwise, the compiler warns that we have not
+ * handled that enum value.
+ */
+typedef enum base_jd_event_code
+{
+ /* HW defined exceptions */
+
+ /** Start of HW Non-fault status codes
+ *
+ * @note Obscurely, BASE_JD_EVENT_TERMINATED indicates a real fault,
+ * because the job was hard-stopped
+ */
+ BASE_JD_EVENT_RANGE_HW_NONFAULT_START = 0,
+
+ /* non-fatal exceptions */
+ BASE_JD_EVENT_NOT_STARTED = 0x00, /**< Can't be seen by userspace, treated as 'previous job done' */
+ BASE_JD_EVENT_DONE = 0x01,
+ BASE_JD_EVENT_STOPPED = 0x03, /**< Can't be seen by userspace, becomes TERMINATED, DONE or JOB_CANCELLED */
+ BASE_JD_EVENT_TERMINATED = 0x04, /**< This is actually a fault status code - the job was hard stopped */
+ BASE_JD_EVENT_ACTIVE = 0x08, /**< Can't be seen by userspace, jobs only returned on complete/fail/cancel */
+
+ /** End of HW Non-fault status codes
+ *
+ * @note Obscurely, BASE_JD_EVENT_TERMINATED indicates a real fault,
+ * because the job was hard-stopped
+ */
+ BASE_JD_EVENT_RANGE_HW_NONFAULT_END = 0x40,
+
+ /** Start of HW fault and SW Error status codes */
+ BASE_JD_EVENT_RANGE_HW_FAULT_OR_SW_ERROR_START = 0x40,
+
+ /* job exceptions */
+ BASE_JD_EVENT_JOB_CONFIG_FAULT = 0x40,
+ BASE_JD_EVENT_JOB_POWER_FAULT = 0x41,
+ BASE_JD_EVENT_JOB_READ_FAULT = 0x42,
+ BASE_JD_EVENT_JOB_WRITE_FAULT = 0x43,
+ BASE_JD_EVENT_JOB_AFFINITY_FAULT = 0x44,
+ BASE_JD_EVENT_JOB_BUS_FAULT = 0x48,
+ BASE_JD_EVENT_INSTR_INVALID_PC = 0x50,
+ BASE_JD_EVENT_INSTR_INVALID_ENC = 0x51,
+ BASE_JD_EVENT_INSTR_TYPE_MISMATCH = 0x52,
+ BASE_JD_EVENT_INSTR_OPERAND_FAULT = 0x53,
+ BASE_JD_EVENT_INSTR_TLS_FAULT = 0x54,
+ BASE_JD_EVENT_INSTR_BARRIER_FAULT = 0x55,
+ BASE_JD_EVENT_INSTR_ALIGN_FAULT = 0x56,
+ BASE_JD_EVENT_DATA_INVALID_FAULT = 0x58,
+ BASE_JD_EVENT_TILE_RANGE_FAULT = 0x59,
+ BASE_JD_EVENT_STATE_FAULT = 0x5A,
+ BASE_JD_EVENT_OUT_OF_MEMORY = 0x60,
+ BASE_JD_EVENT_UNKNOWN = 0x7F,
+
+ /* GPU exceptions */
+ BASE_JD_EVENT_DELAYED_BUS_FAULT = 0x80,
+ BASE_JD_EVENT_SHAREABILITY_FAULT = 0x88,
+
+ /* MMU exceptions */
+ BASE_JD_EVENT_TRANSLATION_FAULT_LEVEL1 = 0xC1,
+ BASE_JD_EVENT_TRANSLATION_FAULT_LEVEL2 = 0xC2,
+ BASE_JD_EVENT_TRANSLATION_FAULT_LEVEL3 = 0xC3,
+ BASE_JD_EVENT_TRANSLATION_FAULT_LEVEL4 = 0xC4,
+ BASE_JD_EVENT_PERMISSION_FAULT = 0xC8,
+ BASE_JD_EVENT_TRANSTAB_BUS_FAULT_LEVEL1 = 0xD1,
+ BASE_JD_EVENT_TRANSTAB_BUS_FAULT_LEVEL2 = 0xD2,
+ BASE_JD_EVENT_TRANSTAB_BUS_FAULT_LEVEL3 = 0xD3,
+ BASE_JD_EVENT_TRANSTAB_BUS_FAULT_LEVEL4 = 0xD4,
+ BASE_JD_EVENT_ACCESS_FLAG = 0xD8,
+
+ /* SW defined exceptions */
+ BASE_JD_EVENT_MEM_GROWTH_FAILED = BASE_JD_SW_EVENT | BASE_JD_SW_EVENT_JOB | 0x000,
+ BASE_JD_EVENT_TIMED_OUT = BASE_JD_SW_EVENT | BASE_JD_SW_EVENT_JOB | 0x001,
+ BASE_JD_EVENT_JOB_CANCELLED = BASE_JD_SW_EVENT | BASE_JD_SW_EVENT_JOB | 0x002,
+ BASE_JD_EVENT_BAG_INVALID = BASE_JD_SW_EVENT | BASE_JD_SW_EVENT_BAG | 0x003,
+
+ /** End of HW fault and SW Error status codes */
+ BASE_JD_EVENT_RANGE_HW_FAULT_OR_SW_ERROR_END = BASE_JD_SW_EVENT | BASE_JD_SW_EVENT_RESERVED | 0x3FF,
+
+ /** Start of SW Success status codes */
+ BASE_JD_EVENT_RANGE_SW_SUCCESS_START = BASE_JD_SW_EVENT | BASE_JD_SW_EVENT_SUCCESS | 0x000,
+
+ BASE_JD_EVENT_PROGRESS_REPORT = BASE_JD_SW_EVENT | BASE_JD_SW_EVENT_SUCCESS | BASE_JD_SW_EVENT_JOB | 0x000,
+ BASE_JD_EVENT_BAG_DONE = BASE_JD_SW_EVENT | BASE_JD_SW_EVENT_SUCCESS | BASE_JD_SW_EVENT_BAG | 0x000,
+ BASE_JD_EVENT_DRV_TERMINATED = BASE_JD_SW_EVENT | BASE_JD_SW_EVENT_SUCCESS | BASE_JD_SW_EVENT_INFO | 0x000,
+
+ /** End of SW Success status codes */
+ BASE_JD_EVENT_RANGE_SW_SUCCESS_END = BASE_JD_SW_EVENT | BASE_JD_SW_EVENT_SUCCESS | BASE_JD_SW_EVENT_RESERVED | 0x3FF,
+
+ /** Start of Kernel-only status codes. Such codes are never returned to user-space */
+ BASE_JD_EVENT_RANGE_KERNEL_ONLY_START = BASE_JD_SW_EVENT | BASE_JD_SW_EVENT_KERNEL | 0x000,
+ BASE_JD_EVENT_REMOVED_FROM_NEXT = BASE_JD_SW_EVENT | BASE_JD_SW_EVENT_KERNEL | BASE_JD_SW_EVENT_JOB | 0x000,
+
+ /** End of Kernel-only status codes. */
+ BASE_JD_EVENT_RANGE_KERNEL_ONLY_END = BASE_JD_SW_EVENT | BASE_JD_SW_EVENT_KERNEL | BASE_JD_SW_EVENT_RESERVED | 0x3FF
+} base_jd_event_code;
+
+/**
+ * @brief Event reporting structure
+ *
+ * This structure is used by the kernel driver to report information
+ * about GPU events. The can either be HW-specific events or low-level
+ * SW events, such as job-chain completion.
+ *
+ * The event code contains an event type field which can be extracted
+ * by ANDing with ::BASE_JD_SW_EVENT_TYPE_MASK.
+ *
+ * Based on the event type base_jd_event::data holds:
+ * @li ::BASE_JD_SW_EVENT_JOB : the offset in the ring-buffer for the completed
+ * job-chain
+ * @li ::BASE_JD_SW_EVENT_BAG : The address of the ::base_jd_bag that has
+ * been completed (ie all contained job-chains have been completed).
+ * @li ::BASE_JD_SW_EVENT_INFO : base_jd_event::data not used
+ */
+typedef struct base_jd_event
+{
+ base_jd_event_code event_code; /**< event code */
+ void * data; /**< event specific data */
+} base_jd_event;
+
+/**
+ * @brief Structure for BASE_JD_REQ_SOFT_DUMP_CPU_GPU_COUNTERS jobs.
+ *
+ * This structure is stored into the memory pointed to by the @c jc field of @ref base_jd_atom.
+ */
+typedef struct base_dump_cpu_gpu_counters {
+ u64 system_time;
+ u64 cycle_counter;
+ u64 sec;
+ u32 usec;
+} base_dump_cpu_gpu_counters;
+
+/** @} end group base_user_api_job_dispatch */
+
+
+#ifdef __KERNEL__
+/*
+ * The following typedefs should be removed when a midg types header is added.
+ * See MIDCOM-1657 for details.
+ */
+typedef u32 midg_product_id;
+typedef u32 midg_cache_features;
+typedef u32 midg_tiler_features;
+typedef u32 midg_mem_features;
+typedef u32 midg_mmu_features;
+typedef u32 midg_js_features;
+typedef u32 midg_as_present;
+typedef u32 midg_js_present;
+
+#define MIDG_MAX_JOB_SLOTS 16
+
+#else
+#include <midg/mali_midg.h>
+#endif
+
+/**
+ * @page page_base_user_api_gpuprops User-side Base GPU Property Query API
+ *
+ * The User-side Base GPU Property Query API encapsulates two
+ * sub-modules:
+ *
+ * - @ref base_user_api_gpuprops_dyn "Dynamic GPU Properties"
+ * - @ref base_plat_config_gpuprops "Base Platform Config GPU Properties"
+ *
+ * There is a related third module outside of Base, which is owned by the MIDG
+ * module:
+ * - @ref midg_gpuprops_static "Midgard Compile-time GPU Properties"
+ *
+ * Base only deals with properties that vary between different Midgard
+ * implementations - the Dynamic GPU properties and the Platform Config
+ * properties.
+ *
+ * For properties that are constant for the Midgard Architecture, refer to the
+ * MIDG module. However, we will discuss their relevance here <b>just to
+ * provide background information.</b>
+ *
+ * @section sec_base_user_api_gpuprops_about About the GPU Properties in Base and MIDG modules
+ *
+ * The compile-time properties (Platform Config, Midgard Compile-time
+ * properties) are exposed as pre-processor macros.
+ *
+ * Complementing the compile-time properties are the Dynamic GPU
+ * Properties, which act as a conduit for the Midgard Configuration
+ * Discovery.
+ *
+ * In general, the dynamic properties are present to verify that the platform
+ * has been configured correctly with the right set of Platform Config
+ * Compile-time Properties.
+ *
+ * As a consistant guide across the entire DDK, the choice for dynamic or
+ * compile-time should consider the following, in order:
+ * -# Can the code be written so that it doesn't need to know the
+ * implementation limits at all?
+ * -# If you need the limits, get the information from the Dynamic Property
+ * lookup. This should be done once as you fetch the context, and then cached
+ * as part of the context data structure, so it's cheap to access.
+ * -# If there's a clear and arguable inefficiency in using Dynamic Properties,
+ * then use a Compile-Time Property (Platform Config, or Midgard Compile-time
+ * property). Examples of where this might be sensible follow:
+ * - Part of a critical inner-loop
+ * - Frequent re-use throughout the driver, causing significant extra load
+ * instructions or control flow that would be worthwhile optimizing out.
+ *
+ * We cannot provide an exhaustive set of examples, neither can we provide a
+ * rule for every possible situation. Use common sense, and think about: what
+ * the rest of the driver will be doing; how the compiler might represent the
+ * value if it is a compile-time constant; whether an OEM shipping multiple
+ * devices would benefit much more from a single DDK binary, instead of
+ * insignificant micro-optimizations.
+ *
+ * @section sec_base_user_api_gpuprops_dyn Dynamic GPU Properties
+ *
+ * Dynamic GPU properties are presented in two sets:
+ * -# the commonly used properties in @ref base_gpu_props, which have been
+ * unpacked from GPU register bitfields.
+ * -# The full set of raw, unprocessed properties in @ref midg_raw_gpu_props
+ * (also a member of @ref base_gpu_props). All of these are presented in
+ * the packed form, as presented by the GPU registers themselves.
+ *
+ * @usecase The raw properties in @ref midg_raw_gpu_props are necessary to
+ * allow a user of the Mali Tools (e.g. PAT) to determine "Why is this device
+ * behaving differently?". In this case, all information about the
+ * configuration is potentially useful, but it <b>does not need to be processed
+ * by the driver</b>. Instead, the raw registers can be processed by the Mali
+ * Tools software on the host PC.
+ *
+ * The properties returned extend the Midgard Configuration Discovery
+ * registers. For example, GPU clock speed is not specified in the Midgard
+ * Architecture, but is <b>necessary for OpenCL's clGetDeviceInfo() function</b>.
+ *
+ * The GPU properties are obtained by a call to
+ * _mali_base_get_gpu_props(). This simply returns a pointer to a const
+ * base_gpu_props structure. It is constant for the life of a base
+ * context. Multiple calls to _mali_base_get_gpu_props() to a base context
+ * return the same pointer to a constant structure. This avoids cache pollution
+ * of the common data.
+ *
+ * This pointer must not be freed, because it does not point to the start of a
+ * region allocated by the memory allocator; instead, just close the @ref
+ * base_context.
+ *
+ *
+ * @section sec_base_user_api_gpuprops_config Platform Config Compile-time Properties
+ *
+ * The Platform Config File sets up gpu properties that are specific to a
+ * certain platform. Properties that are 'Implementation Defined' in the
+ * Midgard Architecture spec are placed here.
+ *
+ * @note Reference configurations are provided for Midgard Implementations, such as
+ * the Mali-T600 family. The customer need not repeat this information, and can select one of
+ * these reference configurations. For example, VA_BITS, PA_BITS and the
+ * maximum number of samples per pixel might vary between Midgard Implementations, but
+ * \b not for platforms using the Mali-T604. This information is placed in
+ * the reference configuration files.
+ *
+ * The System Integrator creates the following structure:
+ * - platform_XYZ
+ * - platform_XYZ/plat
+ * - platform_XYZ/plat/plat_config.h
+ *
+ * They then edit plat_config.h, using the example plat_config.h files as a
+ * guide.
+ *
+ * At the very least, the customer must set @ref CONFIG_GPU_CORE_TYPE, and will
+ * receive a helpful \#error message if they do not do this correctly. This
+ * selects the Reference Configuration for the Midgard Implementation. The rationale
+ * behind this decision (against asking the customer to write \#include
+ * <gpus/mali_t600.h> in their plat_config.h) is as follows:
+ * - This mechanism 'looks' like a regular config file (such as Linux's
+ * .config)
+ * - It is difficult to get wrong in a way that will produce strange build
+ * errors:
+ * - They need not know where the mali_t600.h, other_midg_gpu.h etc. files are stored - and
+ * so they won't accidentally pick another file with 'mali_t600' in its name
+ * - When the build doesn't work, the System Integrator may think the DDK is
+ * doesn't work, and attempt to fix it themselves:
+ * - For the @ref CONFIG_GPU_CORE_TYPE mechanism, the only way to get past the
+ * error is to set @ref CONFIG_GPU_CORE_TYPE, and this is what the \#error tells
+ * you.
+ * - For a \#include mechanism, checks must still be made elsewhere, which the
+ * System Integrator may try working around by setting \#defines (such as
+ * VA_BITS) themselves in their plat_config.h. In the worst case, they may
+ * set the prevention-mechanism \#define of
+ * "A_CORRECT_MIDGARD_CORE_WAS_CHOSEN".
+ * - In this case, they would believe they are on the right track, because
+ * the build progresses with their fix, but with errors elsewhere.
+ *
+ * However, there is nothing to prevent the customer using \#include to organize
+ * their own configurations files hierarchically.
+ *
+ * The mechanism for the header file processing is as follows:
+ *
+ * @dot
+ digraph plat_config_mechanism {
+ rankdir=BT
+ size="6,6"
+
+ "mali_base.h";
+ "midg/midg.h";
+
+ node [ shape=box ];
+ {
+ rank = same; ordering = out;
+
+ "midg/midg_gpu_props.h";
+ "base/midg_gpus/mali_t600.h";
+ "base/midg_gpus/other_midg_gpu.h";
+ }
+ { rank = same; "plat/plat_config.h"; }
+ {
+ rank = same;
+ "midg/midg.h" [ shape=box ];
+ gpu_chooser [ label="" style="invisible" width=0 height=0 fixedsize=true ];
+ select_gpu [ label="Mali-T600 | Other\n(select_gpu.h)" shape=polygon,sides=4,distortion=0.25 width=3.3 height=0.99 fixedsize=true ] ;
+ }
+ node [ shape=box ];
+ { rank = same; "plat/plat_config.h"; }
+ { rank = same; "mali_base.h"; }
+
+
+
+ "mali_base.h" -> "midg/midg.h" -> "midg/midg_gpu_props.h";
+ "mali_base.h" -> "plat/plat_config.h" ;
+ "mali_base.h" -> select_gpu ;
+
+ "plat/plat_config.h" -> gpu_chooser [style="dotted,bold" dir=none weight=4] ;
+ gpu_chooser -> select_gpu [style="dotted,bold"] ;
+
+ select_gpu -> "base/midg_gpus/mali_t600.h" ;
+ select_gpu -> "base/midg_gpus/other_midg_gpu.h" ;
+ }
+ @enddot
+ *
+ *
+ * @section sec_base_user_api_gpuprops_kernel Kernel Operation
+ *
+ * During Base Context Create time, user-side makes a single kernel call:
+ * - A call to fill user memory with GPU information structures
+ *
+ * The kernel-side will fill the provided the entire processed @ref base_gpu_props
+ * structure, because this information is required in both
+ * user and kernel side; it does not make sense to decode it twice.
+ *
+ * Coherency groups must be derived from the bitmasks, but this can be done
+ * kernel side, and just once at kernel startup: Coherency groups must already
+ * be known kernel-side, to support chains that specify a 'Only Coherent Group'
+ * SW requirement, or 'Only Coherent Group with Tiler' SW requirement.
+ *
+ * @section sec_base_user_api_gpuprops_cocalc Coherency Group calculation
+ * Creation of the coherent group data is done at device-driver startup, and so
+ * is one-time. This will most likely involve a loop with CLZ, shifting, and
+ * bit clearing on the L2_PRESENT or L3_PRESENT masks, depending on whether the
+ * system is L2 or L2+L3 Coherent. The number of shader cores is done by a
+ * population count, since faulty cores may be disabled during production,
+ * producing a non-contiguous mask.
+ *
+ * The memory requirements for this algoirthm can be determined either by a u64
+ * population count on the L2/L3_PRESENT masks (a LUT helper already is
+ * requried for the above), or simple assumption that there can be no more than
+ * 16 coherent groups, since core groups are typically 4 cores.
+ */
+
+
+/**
+ * @addtogroup base_user_api_gpuprops User-side Base GPU Property Query APIs
+ * @{
+ */
+
+
+/**
+ * @addtogroup base_user_api_gpuprops_dyn Dynamic HW Properties
+ * @{
+ */
+
+
+#define BASE_GPU_NUM_TEXTURE_FEATURES_REGISTERS 3
+
+#define BASE_MAX_COHERENT_GROUPS 16
+
+struct mali_base_gpu_core_props
+{
+ /**
+ * Product specific value.
+ */
+ midg_product_id product_id;
+
+ /**
+ * Status of the GPU release.
+ * No defined values, but starts at 0 and increases by one for each release
+ * status (alpha, beta, EAC, etc.).
+ * 4 bit values (0-15).
+ */
+ u16 version_status;
+
+ /**
+ * Minor release number of the GPU. "P" part of an "RnPn" release number.
+ * 8 bit values (0-255).
+ */
+ u16 minor_revision;
+
+ /**
+ * Major release number of the GPU. "R" part of an "RnPn" release number.
+ * 4 bit values (0-15).
+ */
+ u16 major_revision;
+
+ /**
+ * @usecase GPU clock speed is not specified in the Midgard Architecture, but is
+ * <b>necessary for OpenCL's clGetDeviceInfo() function</b>.
+ */
+ u32 gpu_speed_mhz;
+
+ /**
+ * @usecase GPU clock max/min speed is required for computing best/worst case
+ * in tasks as job scheduling ant irq_throttling. (It is not specified in the
+ * Midgard Architecture).
+ */
+ u32 gpu_freq_khz_max;
+ u32 gpu_freq_khz_min;
+
+ /**
+ * Size of the shader program counter, in bits.
+ */
+ u32 log2_program_counter_size;
+
+ /**
+ * TEXTURE_FEATURES_x registers, as exposed by the GPU. This is a
+ * bitpattern where a set bit indicates that the format is supported.
+ *
+ * Before using a texture format, it is recommended that the corresponding
+ * bit be checked.
+ */
+ u32 texture_features[BASE_GPU_NUM_TEXTURE_FEATURES_REGISTERS];
+
+ /**
+ * Theoretical maximum memory available to the GPU. It is unlikely that a
+ * client will be able to allocate all of this memory for their own
+ * purposes, but this at least provides an upper bound on the memory
+ * available to the GPU.
+ *
+ * This is required for OpenCL's clGetDeviceInfo() call when
+ * CL_DEVICE_GLOBAL_MEM_SIZE is requested, for OpenCL GPU devices. The
+ * client will not be expecting to allocate anywhere near this value.
+ */
+ u64 gpu_available_memory_size;
+
+ /**
+ * @usecase Version string: For use by glGetString( GL_RENDERER ); (and similar
+ * for other APIs)
+ */
+ const char * version_string;
+};
+
+/**
+ *
+ * More information is possible - but associativity and bus width are not
+ * required by upper-level apis.
+
+ */
+struct mali_base_gpu_cache_props
+{
+ u32 log2_line_size;
+ u32 log2_cache_size;
+};
+
+struct mali_base_gpu_tiler_props
+{
+ u32 bin_size_bytes; /* Max is 4*2^15 */
+ u32 max_active_levels; /* Max is 2^15 */
+};
+
+/**
+ * @brief descriptor for a coherent group
+ *
+ * \c core_mask exposes all cores in that coherent group, and \c num_cores
+ * provides a cached population-count for that mask.
+ *
+ * @note Whilst all cores are exposed in the mask, not all may be available to
+ * the application, depending on the Kernel Job Scheduler policy. Therefore,
+ * the application should not further restrict the core mask itself, as it may
+ * result in an empty core mask. However, it can guarentee that there will be
+ * at least one core available for each core group exposed .
+ *
+ * @usecase Chains marked at certain user-side priorities (e.g. the Long-running
+ * (batch) priority ) can be prevented from running on entire core groups by the
+ * Kernel Chain Scheduler policy.
+ *
+ * @note if u64s must be 8-byte aligned, then this structure has 32-bits of wastage.
+ */
+struct mali_base_gpu_coherent_group
+{
+ u64 core_mask; /**< Core restriction mask required for the group */
+ u16 num_cores; /**< Number of cores in the group */
+};
+
+/**
+ * @brief Coherency group information
+ *
+ * Note that the sizes of the members could be reduced. However, the \c group
+ * member might be 8-byte aligned to ensure the u64 core_mask is 8-byte
+ * aligned, thus leading to wastage if the other members sizes were reduced.
+ *
+ * The groups are sorted by core mask. The core masks are non-repeating and do
+ * not intersect.
+ */
+struct mali_base_gpu_coherent_group_info
+{
+ u32 num_groups;
+
+ /**
+ * Number of core groups (coherent or not) in the GPU. Equivalent to the number of L2 Caches.
+ *
+ * The GPU Counter dumping writes 2048 bytes per core group, regardless of
+ * whether the core groups are coherent or not. Hence this member is needed
+ * to calculate how much memory is required for dumping.
+ *
+ * @note Do not use it to work out how many valid elements are in the
+ * group[] member. Use num_groups instead.
+ */
+ u32 num_core_groups;
+
+ /**
+ * Coherency features of the memory, accessed by @ref midg_mem_features
+ * methods
+ */
+ midg_mem_features coherency;
+
+ /**
+ * Descriptors of coherent groups
+ */
+ struct mali_base_gpu_coherent_group group[BASE_MAX_COHERENT_GROUPS];
+};
+
+
+/**
+ * A complete description of the GPU's Hardware Configuration Discovery
+ * registers.
+ *
+ * The information is presented inefficiently for access. For frequent access,
+ * the values should be better expressed in an unpacked form in the
+ * base_gpu_props structure.
+ *
+ * @usecase The raw properties in @ref midg_raw_gpu_props are necessary to
+ * allow a user of the Mali Tools (e.g. PAT) to determine "Why is this device
+ * behaving differently?". In this case, all information about the
+ * configuration is potentially useful, but it <b>does not need to be processed
+ * by the driver</b>. Instead, the raw registers can be processed by the Mali
+ * Tools software on the host PC.
+ *
+ */
+struct midg_raw_gpu_props
+{
+ u64 shader_present;
+ u64 tiler_present;
+ u64 l2_present;
+ u64 l3_present;
+
+ midg_cache_features l2_features;
+ midg_cache_features l3_features;
+ midg_mem_features mem_features;
+ midg_mmu_features mmu_features;
+
+ midg_as_present as_present;
+
+ u32 js_present;
+ midg_js_features js_features[MIDG_MAX_JOB_SLOTS];
+ midg_tiler_features tiler_features;
+
+ u32 gpu_id;
+};
+
+
+
+/**
+ * Return structure for _mali_base_get_gpu_props().
+ *
+ */
+typedef struct mali_base_gpu_props
+{
+ struct mali_base_gpu_core_props core_props;
+ struct mali_base_gpu_cache_props l2_props;
+ struct mali_base_gpu_cache_props l3_props;
+ struct mali_base_gpu_tiler_props tiler_props;
+
+ /** This member is large, likely to be 128 bytes */
+ struct midg_raw_gpu_props raw_props;
+
+ /** This must be last member of the structure */
+ struct mali_base_gpu_coherent_group_info coherency_info;
+}base_gpu_props;
+
+/** @} end group base_user_api_gpuprops_dyn */
+
+/** @} end group base_user_api_gpuprops */
+
+/**
+ * @addtogroup base_user_api_core User-side Base core APIs
+ * @{
+ */
+
+/**
+ * \enum base_context_create_flags
+ *
+ * Flags to pass to ::base_context_init.
+ * Flags can be ORed together to enable multiple things.
+ *
+ * These share the same space as @ref basep_context_private_flags, and so must
+ * not collide with them.
+ */
+enum base_context_create_flags
+{
+ /** No flags set */
+ BASE_CONTEXT_CREATE_FLAG_NONE = 0,
+
+ /** Base context is embedded in a cctx object (flag used for CINSTR software counter macros) */
+ BASE_CONTEXT_CCTX_EMBEDDED = (1u << 0),
+
+ /** Base context is a 'System Monitor' context for Hardware counters.
+ *
+ * One important side effect of this is that job submission is disabled. */
+ BASE_CONTEXT_SYSTEM_MONITOR_SUBMIT_DISABLED = (1u << 1),
+
+ /** Base context flag indicating a 'hint' that this context uses Compute
+ * Jobs only.
+ *
+ * Specifially, this means that it only sends atoms that <b>do not</b>
+ * contain the following @ref base_jd_corereq :
+ * - BASE_JD_REQ_FS
+ * - BASE_JD_REQ_T
+ *
+ * Violation of these requirements will cause the Job-Chains to be rejected.
+ *
+ * In addition, it is inadvisable for the atom's Job-Chains to contain Jobs
+ * of the following @ref midg_job_type (whilst it may work now, it may not
+ * work in future) :
+ * - @ref MIDG_JOB_VERTEX
+ * - @ref MIDG_JOB_GEOMETRY
+ *
+ * @note An alternative to using this is to specify the BASE_JD_REQ_ONLY_COMPUTE
+ * requirement in atoms.
+ */
+ BASE_CONTEXT_HINT_ONLY_COMPUTE = (1u << 2)
+};
+
+/**
+ * Bitpattern describing the ::base_context_create_flags that can be passed to base_context_init()
+ */
+#define BASE_CONTEXT_CREATE_ALLOWED_FLAGS \
+ ( ((u32)BASE_CONTEXT_CCTX_EMBEDDED) | \
+ ((u32)BASE_CONTEXT_SYSTEM_MONITOR_SUBMIT_DISABLED) | \
+ ((u32)BASE_CONTEXT_HINT_ONLY_COMPUTE) )
+
+/**
+ * Bitpattern describing the ::base_context_create_flags that can be passed to the kernel
+ */
+#define BASE_CONTEXT_CREATE_KERNEL_FLAGS \
+ ( ((u32)BASE_CONTEXT_SYSTEM_MONITOR_SUBMIT_DISABLED) | \
+ ((u32)BASE_CONTEXT_HINT_ONLY_COMPUTE) )
+
+
+/**
+ * Private flags used on the base context
+ *
+ * These start at bit 31, and run down to zero.
+ *
+ * They share the same space as @ref base_context_create_flags, and so must
+ * not collide with them.
+ */
+enum basep_context_private_flags
+{
+ /** Private flag tracking whether job descriptor dumping is disabled */
+ BASEP_CONTEXT_FLAG_JOB_DUMP_DISABLED = (1 << 31)
+};
+
+/** @} end group base_user_api_core */
+
+/** @} end group base_user_api */
+
+/**
+ * @addtogroup base_plat_config_gpuprops Base Platform Config GPU Properties
+ * @{
+ *
+ * C Pre-processor macros are exposed here to do with Platform
+ * Config.
+ *
+ * These include:
+ * - GPU Properties that are constant on a particular Midgard Family
+ * Implementation e.g. Maximum samples per pixel on Mali-T600.
+ * - General platform config for the GPU, such as the GPU major and minor
+ * revison.
+ */
+
+/** @} end group base_plat_config_gpuprops */
+
+/**
+ @addtogroup basecpuprops
+ * @{
+ */
+
+/**
+ * @brief CPU Property Flag for base_cpu_props::cpu_flags, indicating a
+ * Little Endian System. If not set in base_cpu_props::cpu_flags, then the
+ * system is Big Endian.
+ *
+ * The compile-time equivalent is @ref CONFIG_CPU_LITTLE_ENDIAN.
+ */
+#define BASE_CPU_PROPERTY_FLAG_LITTLE_ENDIAN F_BIT_0
+
+/** @brief Platform Dynamic CPU properties structure */
+typedef struct base_cpu_props {
+ u32 nr_cores; /**< Number of CPU cores */
+
+ /**
+ * CPU page size as a Logarithm to Base 2. The compile-time
+ * equivalent is @ref CONFIG_CPU_PAGE_SIZE_LOG2
+ */
+ u32 cpu_page_size_log2;
+
+ /**
+ * CPU L1 Data cache line size as a Logarithm to Base 2. The compile-time
+ * equivalent is @ref CONFIG_CPU_L1_DCACHE_LINE_SIZE_LOG2.
+ */
+ u32 cpu_l1_dcache_line_size_log2;
+
+ /**
+ * CPU L1 Data cache size, in bytes. The compile-time equivalient is
+ * @ref CONFIG_CPU_L1_DCACHE_SIZE.
+ *
+ * This CPU Property is mainly provided to implement OpenCL's
+ * clGetDeviceInfo(), which allows the CL_DEVICE_GLOBAL_MEM_CACHE_SIZE
+ * hint to be queried.
+ */
+ u32 cpu_l1_dcache_size;
+
+ /**
+ * CPU Property Flags bitpattern.
+ *
+ * This is a combination of bits as specified by the macros prefixed with
+ * 'BASE_CPU_PROPERTY_FLAG_'.
+ */
+ u32 cpu_flags;
+
+ /**
+ * Maximum clock speed in MHz.
+ * @usecase 'Maximum' CPU Clock Speed information is required by OpenCL's
+ * clGetDeviceInfo() function for the CL_DEVICE_MAX_CLOCK_FREQUENCY hint.
+ */
+ u32 max_cpu_clock_speed_mhz;
+
+ /**
+ * @brief Total memory, in bytes.
+ *
+ * This is the theoretical maximum memory available to the CPU. It is
+ * unlikely that a client will be able to allocate all of this memory for
+ * their own purposes, but this at least provides an upper bound on the
+ * memory available to the CPU.
+ *
+ * This is required for OpenCL's clGetDeviceInfo() call when
+ * CL_DEVICE_GLOBAL_MEM_SIZE is requested, for OpenCL CPU devices.
+ */
+ u64 available_memory_size;
+} base_cpu_props;
+/** @} end group basecpuprops */
+
+/** @} end group base_api */
+
+#endif /* _BASE_KERNEL_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_config.h
+ * Configuration API and Attributes for KBase
+ */
+
+#ifndef _KBASE_CONFIG_H_
+#define _KBASE_CONFIG_H_
+
+#include <malisw/mali_stdtypes.h>
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_kbase_api
+ * @{
+ */
+
+/**
+ * @addtogroup kbase_config Configuration API and Attributes
+ * @{
+ */
+
+#if MALI_CUSTOMER_RELEASE == 0
+/* This flag is set for internal builds so we can run tests without credentials. */
+#define KBASE_HWCNT_DUMP_BYPASS_ROOT 1
+#else
+#define KBASE_HWCNT_DUMP_BYPASS_ROOT 0
+#endif
+
+/**
+ * Relative memory performance indicators. Enum elements should always be defined in slowest to fastest order.
+ */
+typedef enum kbase_memory_performance
+{
+ KBASE_MEM_PERF_SLOW,
+ KBASE_MEM_PERF_NORMAL,
+ KBASE_MEM_PERF_FAST,
+
+ KBASE_MEM_PERF_MAX_VALUE = KBASE_MEM_PERF_FAST
+} kbase_memory_performance;
+
+/**
+ * Device wide configuration
+ */
+enum
+{
+ /**
+ * Invalid attribute ID (reserve 0).
+ *
+ * Attached value: Ignored
+ * Default value: NA
+ * */
+ KBASE_CONFIG_ATTR_INVALID,
+
+ /**
+ * Memory resource object.
+ * Multiple resources can be listed.
+ * The resources will be used in the order listed
+ * in the configuration attribute list if they have no other
+ * preferred order based on the memory resource property list
+ * (see ::kbase_memory_attribute).
+ *
+ * Attached value: Pointer to a kbase_memory_resource object.
+ * Default value: No resources
+ * */
+
+ KBASE_CONFIG_ATTR_MEMORY_RESOURCE,
+ /**
+ * Maximum of memory which can be allocated from the OS
+ * to be used by the GPU (shared memory).
+ * This must be greater than 0 as the GPU page tables
+ * are currently stored in a shared memory allocation.
+ *
+ * Attached value: number in bytes
+ * Default value: Limited by available memory
+ */
+ KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_MAX,
+
+ /**
+ * Relative performance for the GPU to access
+ * OS shared memory.
+ *
+ * Attached value: ::kbase_memory_performance member
+ * Default value: ::KBASE_MEM_PERF_NORMAL
+ */
+ KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_PERF_GPU,
+
+ /**
+ * Limit (in bytes) the amount of memory a single process
+ * can allocate across all memory banks (including OS shared memory)
+ * for use by the GPU.
+ *
+ * Attached value: number in bytes
+ * Default value: Limited by available memory
+ */
+ KBASE_CONFIG_ATTR_MEMORY_PER_PROCESS_LIMIT,
+
+ /**
+ * UMP device mapping.
+ * Which UMP device this GPU should be mapped to.
+ *
+ * Attached value: UMP_DEVICE_<device>_SHIFT
+ * Default value: UMP_DEVICE_W_SHIFT
+ */
+ KBASE_CONFIG_ATTR_UMP_DEVICE,
+
+ /**
+ * Maximum frequency GPU will be clocked at. Given in kHz.
+ * This must be specified as there is no default value.
+ *
+ * Attached value: number in kHz
+ * Default value: NA
+ */
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MAX,
+
+ /**
+ * Minimum frequency GPU will be clocked at. Given in kHz.
+ * This must be specified as there is no default value.
+ *
+ * Attached value: number in kHz
+ * Default value: NA
+ */
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MIN,
+
+ /**
+ * Irq throttle. It is the minimum desired time in between two
+ * consecutive gpu interrupts (given in 'us'). The irq throttle
+ * gpu register will be configured after this, taking into
+ * account the configured max frequency.
+ *
+ * Attached value: number in micro seconds
+ * Default value: see DEFAULT_IRQ_THROTTLE_TIME_US
+ */
+ KBASE_CONFIG_ATTR_GPU_IRQ_THROTTLE_TIME_US,
+
+ /*** Begin Job Scheduling Configs ***/
+ /**
+ * Job Scheduler scheduling tick granuality. This is in nanoseconds to
+ * allow HR timer support.
+ *
+ * On each scheduling tick, the scheduler may decide to:
+ * -# soft stop a job (the job will be re-run later, and other jobs will
+ * be able to run on the GPU now). This effectively controls the
+ * 'timeslice' given to a job.
+ * -# hard stop a job (to kill a job if it has spent too long on the GPU
+ * and didn't soft-stop).
+ *
+ * The numbers of ticks for these events are controlled by:
+ * - @ref KBASE_CONIFG_ATTR_JS_SOFT_STOP_TICKS
+ * - @ref KBASE_CONIFG_ATTR_JS_HARD_STOP_TICKS_SS
+ * - @ref KBASE_CONIFG_ATTR_JS_HARD_STOP_TICKS_NSS
+ *
+ * A soft-stopped job will later be resumed, allowing it to use more GPU
+ * time <em>in total</em> than that defined by any of the above. However,
+ * the scheduling policy attempts to limit the amount of \em uninterrupted
+ * time spent on the GPU using the above values (that is, the 'timeslice'
+ * of a job)
+ *
+ * This value is supported by the following scheduling policies:
+ * - The Completely Fair Share (CFS) policy
+ *
+ * Attached value: unsigned 32-bit kbasep_js_device_data::scheduling_tick_ns.
+ * The value might be rounded down to lower precision. Must be non-zero
+ * after rounding.<br>
+ * Default value: @ref DEFAULT_JS_SCHEDULING_TICK_NS
+ *
+ * @note this value is allowed to be greater than
+ * @ref KBASE_CONFIG_ATTR_JS_CTX_TIMESLICE_NS. This allows jobs to run on (much)
+ * longer than the job-timeslice, but once this happens, the context gets
+ * scheduled in (much) less frequently than others that stay within the
+ * ctx-timeslice.
+ */
+ KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS,
+
+ /**
+ * Job Scheduler minimum number of scheduling ticks before jobs are soft-stopped.
+ *
+ * This defines the amount of time a job is allowed to stay on the GPU,
+ * before it is soft-stopped to allow other jobs to run.
+ *
+ * That is, this defines the 'timeslice' of the job. It is separate from the
+ * timeslice of the context that contains the job (see
+ * @ref KBASE_CONFIG_ATTR_JS_CTX_TIMESLICE_NS).
+ *
+ * This value is supported by the following scheduling policies:
+ * - The Completely Fair Share (CFS) policy
+ *
+ * Attached value: unsigned 32-bit kbasep_js_device_data::soft_stop_ticks<br>
+ * Default value: @ref DEFAULT_JS_SOFT_STOP_TICKS
+ *
+ * @note a value of zero means "the quickest time to soft-stop a job",
+ * which is somewhere between instant and one tick later.
+ *
+ * @note this value is allowed to be greater than
+ * @ref KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS or
+ * @ref KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS. This effectively disables
+ * soft-stop, and just uses hard-stop instead. In this case, this value
+ * should be much greater than any of the hard stop values (to avoid
+ * soft-stop-after-hard-stop)
+ *
+ * @see KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS
+ */
+ KBASE_CONFIG_ATTR_JS_SOFT_STOP_TICKS,
+
+ /**
+ * Job Scheduler minimum number of scheduling ticks before Soft-Stoppable
+ * (BASE_JD_REQ_NSS bit \b clear) jobs are hard-stopped.
+ *
+ * This defines the amount of time a Soft-Stoppable job is allowed to spend
+ * on the GPU before it is killed. Such jobs won't be resumed if killed.
+ *
+ * This value is supported by the following scheduling policies:
+ * - The Completely Fair Share (CFS) policy
+ *
+ * Attached value: unsigned 32-bit kbasep_js_device_data::hard_stop_ticks_ss<br>
+ * Default value: @ref DEFAULT_JS_HARD_STOP_TICKS_SS
+ *
+ * @note a value of zero means "the quickest time to hard-stop a job",
+ * which is somewhere between instant and one tick later.
+ *
+ * @see KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS
+ */
+ KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS,
+
+ /**
+ * Job Scheduler minimum number of scheduling ticks before Non-Soft-Stoppable
+ * (BASE_JD_REQ_NSS bit \b set) jobs are hard-stopped.
+ *
+ * This defines the amount of time a Non-Soft-Stoppable job is allowed to spend
+ * on the GPU before it is killed. Such jobs won't be resumed if killed.
+ *
+ * This value is supported by the following scheduling policies:
+ * - The Completely Fair Share (CFS) policy
+ *
+ * Attached value: unsigned 32-bit kbasep_js_device_data::hard_stop_ticks_nss<br>
+ * Default value: @ref DEFAULT_JS_HARD_STOP_TICKS_NSS
+ *
+ * @note a value of zero means "the quickest time to hard-stop a job",
+ * which is somewhere between instant and one tick later.
+ *
+ * @see KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS
+ */
+ KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS,
+
+ /**
+ * Job Scheduler timeslice that a context is scheduled in for, in nanoseconds.
+ *
+ * When a context has used up this amount of time across its jobs, it is
+ * scheduled out to let another run.
+ *
+ * @note the resolution is nanoseconds (ns) here, because that's the format
+ * often used by the OS.
+ *
+ * This value controls affects the actual time defined by the following
+ * config values:
+ * - @ref KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_INIT_SLICES
+ * - @ref KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_MIN_SLICES
+ *
+ * This value is supported by the following scheduling policies:
+ * - The Completely Fair Share (CFS) policy
+ *
+ * Attached value: unsigned 32-bit kbasep_js_device_data::ctx_timeslice_ns.
+ * The value might be rounded down to lower precision.<br>
+ * Default value: @ref DEFAULT_JS_CTX_TIMESLICE_NS
+ *
+ * @note a value of zero models a "Round Robin" scheduling policy, and
+ * disables @ref KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_INIT_SLICES
+ * (initially causing LIFO scheduling) and
+ * @ref KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_MIN_SLICES (allowing
+ * not-run-often contexts to get scheduled in quickly, but to only use
+ * a single timeslice when they get scheduled in).
+ */
+ KBASE_CONFIG_ATTR_JS_CTX_TIMESLICE_NS,
+
+ /**
+ * Job Scheduler initial runtime of a context for the CFS Policy, in time-slices.
+ *
+ * This value is relative to that of the least-run context, and defines
+ * where in the CFS queue a new context is added. A value of 1 means 'after
+ * the least-run context has used its timeslice'. Therefore, when all
+ * contexts consistently use the same amount of time, a value of 1 models a
+ * FIFO. A value of 0 would model a LIFO.
+ *
+ * The value is represented in "numbers of time slices". Multiply this
+ * value by that defined in @ref KBASE_CONFIG_ATTR_JS_CTX_TIMESLICE_NS to get
+ * the time value for this in nanoseconds.
+ *
+ * Attached value: unsigned 32-bit kbasep_js_device_data::cfs_ctx_runtime_init_slices<br>
+ * Default value: @ref DEFAULT_JS_CFS_CTX_RUNTIME_INIT_SLICES
+ */
+ KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_INIT_SLICES,
+
+ /**
+ * Job Scheduler minimum runtime value of a context for CFS, in time_slices
+ * relative to that of the least-run context.
+ *
+ * This is a measure of how much preferrential treatment is given to a
+ * context that is not run very often.
+ *
+ * Specficially, this value defines how many timeslices such a context is
+ * (initially) allowed to use at once. Such contexts (e.g. 'interactive'
+ * processes) will appear near the front of the CFS queue, and can initially
+ * use more time than contexts that run continuously (e.g. 'batch'
+ * processes).
+ *
+ * This limit \b prevents a "stored-up timeslices" DoS attack, where a ctx
+ * not run for a long time attacks the system by using a very large initial
+ * number of timeslices when it finally does run.
+ *
+ * Attached value: unsigned 32-bit kbasep_js_device_data::cfs_ctx_runtime_min_slices<br>
+ * Default value: @ref DEFAULT_JS_CFS_CTX_RUNTIME_MIN_SLICES
+ *
+ * @note A value of zero allows not-run-often contexts to get scheduled in
+ * quickly, but to only use a single timeslice when they get scheduled in.
+ */
+ KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_MIN_SLICES,
+
+ /**
+ * Job Scheduler minimum number of scheduling ticks before Soft-Stoppable
+ * (BASE_JD_REQ_NSS bit \b clear) jobs cause the GPU to be reset.
+ *
+ * This defines the amount of time a Soft-Stoppable job is allowed to spend
+ * on the GPU before it is assumed that the GPU has hung and needs to be reset.
+ * The assumes that the job has been hard-stopped already and so the presence of
+ * a job that has remained on the GPU for so long indicates that the GPU has in some
+ * way hung.
+ *
+ * This value is supported by the following scheduling policies:
+ * - The Completely Fair Share (CFS) policy
+ *
+ * Attached value: unsigned 32-bit kbasep_js_device_data::gpu_reset_ticks_nss<br>
+ * Default value: @ref DEFAULT_JS_RESET_TICKS_SS
+ *
+ * @see KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS
+ */
+ KBASE_CONFIG_ATTR_JS_RESET_TICKS_SS,
+
+ /**
+ * Job Scheduler minimum number of scheduling ticks before Non-Soft-Stoppable
+ * (BASE_JD_REQ_NSS bit \b set) jobs cause the GPU to be reset.
+ *
+ * This defines the amount of time a Non-Soft-Stoppable job is allowed to spend
+ * on the GPU before it is assumed that the GPU has hung and needs to be reset.
+ * The assumes that the job has been hard-stopped already and so the presence of
+ * a job that has remained on the GPU for so long indicates that the GPU has in some
+ * way hung.
+ *
+ * This value is supported by the following scheduling policies:
+ * - The Completely Fair Share (CFS) policy
+ *
+ * Attached value: unsigned 32-bit kbasep_js_device_data::gpu_reset_ticks_nss<br>
+ * Default value: @ref DEFAULT_JS_RESET_TICKS_NSS
+ *
+ * @see KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS
+ */
+ KBASE_CONFIG_ATTR_JS_RESET_TICKS_NSS,
+
+ /**
+ * Number of milliseconds given for other jobs on the GPU to be
+ * soft-stopped when the GPU needs to be reset.
+ *
+ * Attached value: number in milliseconds
+ * Default value: @ref DEFAULT_JS_RESET_TIMEOUT_MS
+ */
+ KBASE_CONFIG_ATTR_JS_RESET_TIMEOUT_MS,
+ /*** End Job Scheduling Configs ***/
+
+ /** Power management configuration
+ *
+ * Attached value: pointer to @ref kbase_pm_callback_conf
+ * Default value: See @ref kbase_pm_callback_conf
+ */
+ KBASE_CONFIG_ATTR_POWER_MANAGEMENT_CALLBACKS,
+
+ /**
+ * Boolean indicating whether the driver is configured to be secure at
+ * a potential loss of performance.
+ *
+ * This currently affects only r0p0-15dev0 HW and earlier.
+ *
+ * On r0p0-15dev0 HW and earlier, there are tradeoffs between security and
+ * performance:
+ *
+ * - When this is set to MALI_TRUE, the driver remains fully secure,
+ * but potentially loses performance compared with setting this to
+ * MALI_FALSE.
+ * - When set to MALI_FALSE, the driver is open to certain security
+ * attacks.
+ *
+ * From r0p0-00rel0 and onwards, there is no security loss by setting
+ * this to MALI_FALSE, and no performance loss by setting it to
+ * MALI_TRUE.
+ *
+ * Attached value: mali_bool value
+ * Default value: @ref DEFAULT_SECURE_BUT_LOSS_OF_PERFORMANCE
+ */
+ KBASE_CONFIG_ATTR_SECURE_BUT_LOSS_OF_PERFORMANCE,
+
+ /**
+ * A pointer to a function that calculates the CPU clock
+ * speed of the platform in MHz - see
+ * @ref kbase_cpuprops_clock_speed_function for the function
+ * prototype.
+ *
+ * Attached value: A @ref kbase_cpuprops_clock_speed_function.
+ * Default Value: Pointer to @ref DEFAULT_CPU_SPEED_FUNC -
+ * returns a clock speed of 100 MHz.
+ */
+ KBASE_CONFIG_ATTR_CPU_SPEED_FUNC,
+
+ /**
+ * Platform specific configuration functions
+ *
+ * Attached value: pointer to @ref kbase_platform_funcs_conf
+ * Default value: See @ref kbase_platform_funcs_conf
+ */
+ KBASE_CONFIG_ATTR_PLATFORM_FUNCS,
+
+ /**
+ * End of attribute list indicator.
+ * The configuration loader will stop processing any more elements
+ * when it encounters this attribute.
+ *
+ * Attached value: Ignored
+ * Default value: NA
+ */
+ KBASE_CONFIG_ATTR_END = 0x1FFFUL
+};
+
+enum
+{
+ /**
+ * Invalid attribute ID (reserve 0).
+ *
+ * Attached value: Ignored
+ * Default value: NA
+ */
+ KBASE_MEM_ATTR_INVALID,
+
+ /**
+ * Relative performance for the CPU to access
+ * the memory resource.
+ *
+ * Attached value: ::kbase_memory_performance member
+ * Default value: ::KBASE_MEM_PERF_NORMAL
+ */
+ KBASE_MEM_ATTR_PERF_CPU,
+
+ /**
+ * Relative performance for the GPU to access
+ * the memory resource.
+ *
+ * Attached value: ::kbase_memory_performance member
+ * Default value: ::KBASE_MEM_PERF_NORMAL
+ */
+ KBASE_MEM_ATTR_PERF_GPU,
+
+ /**
+ * End of attribute list indicator.
+ * The memory resource loader will stop processing any more
+ * elements when it encounters this attribute.
+ *
+ * Attached value: Ignored
+ * Default value: NA
+ */
+ KBASE_MEM_ATTR_END = 0x1FFFUL
+};
+
+
+/*
+ * @brief specifies a single attribute
+ *
+ * Attribute is identified by attr field. Data is either integer or a pointer to attribute-specific structure.
+ */
+typedef struct kbase_attribute
+{
+ int id;
+ uintptr_t data;
+} kbase_attribute;
+
+/*
+ * @brief Specifies dedicated memory bank
+ *
+ * Specifies base, size and attributes of a memory bank
+ */
+typedef struct kbase_memory_resource
+{
+ u64 base;
+ u64 size;
+ struct kbase_attribute * attributes;
+ const char * name;
+} kbase_memory_resource;
+
+/* Forward declaration of kbase_device */
+struct kbase_device;
+
+/*
+ * @brief Specifies the functions for platform specific initialization and termination
+ *
+ * By default no functions are required. No additional platform specific control is necessary.
+ */
+typedef struct kbase_platform_funcs_conf
+{
+ /**
+ * Function pointer for platform specific initialization or NULL if no initialization function is required.
+ * This function will be called \em before any other callbacks listed in the kbase_attribute struct (such as
+ * Power Management callbacks).
+ * The platform specific private pointer kbase_device::platform_context can be accessed (and possibly initialized) in here.
+ */
+ mali_bool (*platform_init_func)(struct kbase_device *kbdev);
+ /**
+ * Function pointer for platform specific termination or NULL if no termination function is required.
+ * This function will be called \em after any other callbacks listed in the kbase_attribute struct (such as
+ * Power Management callbacks).
+ * The platform specific private pointer kbase_device::platform_context can be accessed (and possibly terminated) in here.
+ */
+ void (*platform_term_func)(struct kbase_device *kbdev);
+
+} kbase_platform_funcs_conf;
+
+/*
+ * @brief Specifies the callbacks for power management
+ *
+ * By default no callbacks will be made and the GPU must not be powered off.
+ */
+typedef struct kbase_pm_callback_conf
+{
+ /** Callback for when the GPU is idle and the power to it can be switched off.
+ *
+ * The system integrator can decide whether to either do nothing, just switch off
+ * the clocks to the GPU, or to completely power down the GPU.
+ * The platform specific private pointer kbase_device::platform_context can be accessed and modified in here. It is the
+ * platform \em callbacks responsiblity to initialize and terminate this pointer if used (see @ref kbase_platform_funcs_conf).
+ */
+ void (*power_off_callback)(struct kbase_device *kbdev);
+
+ /** Callback for when the GPU is about to become active and power must be supplied.
+ *
+ * This function must not return until the GPU is powered and clocked sufficiently for register access to
+ * succeed. The return value specifies whether the GPU was powered down since the call to power_off_callback.
+ * If the GPU state has been lost then this function must return 1, otherwise it should return 0.
+ * The platform specific private pointer kbase_device::platform_context can be accessed and modified in here. It is the
+ * platform \em callbacks responsiblity to initialize and terminate this pointer if used (see @ref kbase_platform_funcs_conf).
+ *
+ * The return value of the first call to this function is ignored.
+ *
+ * @return 1 if the GPU state may have been lost, 0 otherwise.
+ */
+ int (*power_on_callback)(struct kbase_device *kbdev);
+} kbase_pm_callback_conf;
+
+/**
+ * Type of the function pointer for KBASE_CONFIG_ATTR_CPU_SPEED_FUNC.
+ *
+ * @param clock_speed [out] Once called this will contain the current CPU clock speed in MHz.
+ * This is mainly used to implement OpenCL's clGetDeviceInfo().
+ *
+ * @return 0 on success, 1 on error.
+ */
+typedef int (*kbase_cpuprops_clock_speed_function)(u32 *clock_speed);
+
+#if !MALI_LICENSE_IS_GPL || (defined(MALI_FAKE_PLATFORM_DEVICE) && MALI_FAKE_PLATFORM_DEVICE)
+/*
+ * @brief Specifies start and end of I/O memory region.
+ */
+typedef struct kbase_io_memory_region
+{
+ u64 start;
+ u64 end;
+} kbase_io_memory_region;
+
+/*
+ * @brief Specifies I/O related resources like IRQs and memory region for I/O operations.
+ */
+typedef struct kbase_io_resources
+{
+ u32 job_irq_number;
+ u32 mmu_irq_number;
+ u32 gpu_irq_number;
+ kbase_io_memory_region io_memory_region;
+} kbase_io_resources;
+
+typedef struct kbase_platform_config
+{
+ const kbase_attribute *attributes;
+ const kbase_io_resources *io_resources;
+ u32 midgard_type;
+} kbase_platform_config;
+
+#endif /* !MALI_LICENSE_IS_GPL || (defined(MALI_FAKE_PLATFORM_DEVICE) && MALI_FAKE_PLATFORM_DEVICE) */
+/**
+ * @brief Return character string associated with the given midgard type.
+ *
+ * @param[in] midgard_type - ID of midgard type
+ *
+ * @return Pointer to NULL-terminated character array associated with the given midgard type
+ */
+const char *kbasep_midgard_type_to_string(u32 midgard_type);
+
+/**
+ * @brief Gets the count of attributes in array
+ *
+ * Function gets the count of attributes in array. Note that end of list indicator is also included.
+ *
+ * @param[in] attributes Array of attributes
+ *
+ * @return Number of attributes in the array including end of list indicator.
+ */
+int kbasep_get_config_attribute_count(const kbase_attribute *attributes);
+
+/**
+ * @brief Gets the count of attributes with specified id
+ *
+ * Function gets the count of attributes with specified id in the given attribute array
+ *
+ * @param[in] attributes Array of attributes
+ * @param[in] attibute_id Id of attributes to count
+ *
+ * @return Number of attributes in the array that have specified id
+ */
+int kbasep_get_config_attribute_count_by_id(const kbase_attribute *attributes, int attribute_id);
+
+/**
+ * @brief Gets the next config attribute with the specified ID from the array of attributes.
+ *
+ * Function gets the next attribute with specified attribute id within specified array. If no such attribute is found,
+ * NULL is returned.
+ *
+ * @param[in] attributes Array of attributes in which lookup is performed
+ * @param[in] attribute_id ID of attribute
+ *
+ * @return Pointer to the first attribute matching id or NULL if none is found.
+ */
+const kbase_attribute *kbasep_get_next_attribute(const kbase_attribute *attributes, int attribute_id);
+
+/**
+ * @brief Gets the value of a single config attribute.
+ *
+ * Function gets the value of attribute specified as parameter. If no such attribute is found in the array of
+ * attributes, default value is used.
+ *
+ * @param[in] kbdev Kbase device pointer
+ * @param[in] attributes Array of attributes in which lookup is performed
+ * @param[in] attribute_id ID of attribute
+ *
+ * @return Value of attribute with the given id
+ */
+uintptr_t kbasep_get_config_value(struct kbase_device *kbdev, const kbase_attribute *attributes, int attribute_id);
+
+/**
+ * @brief Obtain memory performance values from kbase_memory_resource structure.
+ *
+ * Function gets cpu and gpu memory performance values from memory resource structure and puts them in the variables
+ * provided as parameters. If the performance of memory bank is not in resource attributes, default value is used.
+ *
+ * @param[in] resource Structure containing information about memory bank to use
+ * @param[out] cpu_performance Pointer to variable which will hold CPU performance value
+ * @param[out] gpu_performance Pointer to variable which will hold GPU performance value
+ */
+void kbasep_get_memory_performance(const kbase_memory_resource *resource,
+ kbase_memory_performance *cpu_performance, kbase_memory_performance *gpu_performance);
+
+/**
+ * @brief Validates configuration attributes
+ *
+ * Function checks validity of given configuration attributes. It will fail on any attribute with unknown id, attribute
+ * with invalid value or attribute list that is not correctly terminated. It will also fail if
+ * KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MIN or KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MAX are not specified.
+ *
+ * @param[in] kbdev Kbase device pointer
+ * @param[in] attributes Array of attributes to validate
+ *
+ * @return MALI_TRUE if no errors have been found in the config. MALI_FALSE otherwise.
+ */
+mali_bool kbasep_validate_configuration_attributes(struct kbase_device *kbdev, const kbase_attribute *attributes);
+
+#if !MALI_LICENSE_IS_GPL || (defined(MALI_FAKE_PLATFORM_DEVICE) && MALI_FAKE_PLATFORM_DEVICE)
+/**
+ * @brief Gets the pointer to platform config.
+ *
+ * @return Pointer to the platform config
+ */
+kbase_platform_config *kbasep_get_platform_config(void);
+#endif /* !MALI_LICENSE_IS_GPL || (defined(MALI_FAKE_PLATFORM_DEVICE) && MALI_FAKE_PLATFORM_DEVICE) */
+
+/**
+ * @brief Platform specific call to initialize hardware
+ *
+ * Function calls a platform defined routine if specified in the configuration attributes.
+ * The routine can initialize any hardware and context state that is required for the GPU block to function.
+ *
+ * @param[in] kbdev Kbase device pointer
+ *
+ * @return MALI_TRUE if no errors have been found in the config. MALI_FALSE otherwise.
+ */
+mali_bool kbasep_platform_device_init(struct kbase_device *kbdev);
+
+/**
+ * @brief Platform specific call to terminate hardware
+ *
+ * Function calls a platform defined routine if specified in the configuration attributes.
+ * The routine can destroy any platform specific context state and shut down any hardware functionality that are
+ * outside of the Power Management callbacks.
+ *
+ * @param[in] kbdev Kbase device pointer
+ *
+ */
+void kbasep_platform_device_term(struct kbase_device *kbdev);
+
+
+/** @} */ /* end group kbase_config */
+/** @} */ /* end group base_kbase_api */
+/** @} */ /* end group base_api */
+
+#endif /* _KBASE_CONFIG_H_ */
--- /dev/null
+/*
+ * Copyright:
+ * ----------------------------------------------------------------------------
+ * This confidential and proprietary software may be used only as authorized
+ * by a licensing agreement from ARM Limited.
+ * (C) COPYRIGHT 2007-2011 ARM Limited , ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorized copies and
+ * copies may only be made to the extent permitted by a licensing agreement
+ * from ARM Limited.
+ * ----------------------------------------------------------------------------
+ */
+
+/**
+ * @addtogroup malisw
+ * @{
+ */
+
+/* ============================================================================
+ Description
+============================================================================ */
+/**
+ * @defgroup arm_cstd_coding_standard ARM C standard types and constants
+ * The common files are a set of standard headers which are used by all parts
+ * of this development, describing types, and generic constants.
+ *
+ * Files in group:
+ * - arm_cstd.h
+ * - arm_cstd_compilers.h
+ * - arm_cstd_types.h
+ * - arm_cstd_types_rvct.h
+ * - arm_cstd_types_gcc.h
+ * - arm_cstd_types_msvc.h
+ * - arm_cstd_pack_push.h
+ * - arm_cstd_pack_pop.h
+ */
+
+/**
+ * @addtogroup arm_cstd_coding_standard
+ * @{
+ */
+
+#ifndef _ARM_CSTD_
+#define _ARM_CSTD_
+
+/* ============================================================================
+ Import standard C99 types
+============================================================================ */
+#include "arm_cstd_compilers.h"
+#include "arm_cstd_types.h"
+
+/* ============================================================================
+ Min and Max Values
+============================================================================ */
+#if !defined(INT8_MAX)
+ #define INT8_MAX ((int8_t) 0x7F)
+#endif
+#if !defined(INT8_MIN)
+ #define INT8_MIN (-INT8_MAX - 1)
+#endif
+
+#if !defined(INT16_MAX)
+ #define INT16_MAX ((int16_t)0x7FFF)
+#endif
+#if !defined(INT16_MIN)
+ #define INT16_MIN (-INT16_MAX - 1)
+#endif
+
+#if !defined(INT32_MAX)
+ #define INT32_MAX ((int32_t)0x7FFFFFFF)
+#endif
+#if !defined(INT32_MIN)
+ #define INT32_MIN (-INT32_MAX - 1)
+#endif
+
+#if !defined(INT64_MAX)
+ #define INT64_MAX ((int64_t)0x7FFFFFFFFFFFFFFFLL)
+#endif
+#if !defined(INT64_MIN)
+ #define INT64_MIN (-INT64_MAX - 1)
+#endif
+
+#if !defined(UINT8_MAX)
+ #define UINT8_MAX ((uint8_t) 0xFF)
+#endif
+
+#if !defined(UINT16_MAX)
+ #define UINT16_MAX ((uint16_t)0xFFFF)
+#endif
+
+#if !defined(UINT32_MAX)
+ #define UINT32_MAX ((uint32_t)0xFFFFFFFF)
+#endif
+
+#if !defined(UINT64_MAX)
+ #define UINT64_MAX ((uint64_t)0xFFFFFFFFFFFFFFFFULL)
+#endif
+
+/* fallbacks if limits.h wasn't available */
+#if !defined(UCHAR_MAX)
+ #define UCHAR_MAX ((unsigned char)~0U)
+#endif
+
+#if !defined(SCHAR_MAX)
+ #define SCHAR_MAX ((signed char)(UCHAR_MAX >> 1))
+#endif
+#if !defined(SCHAR_MIN)
+ #define SCHAR_MIN ((signed char)(-SCHAR_MAX - 1))
+#endif
+
+#if !defined(USHRT_MAX)
+ #define USHRT_MAX ((unsigned short)~0U)
+#endif
+
+#if !defined(SHRT_MAX)
+ #define SHRT_MAX ((signed short)(USHRT_MAX >> 1))
+#endif
+#if !defined(SHRT_MIN)
+ #define SHRT_MIN ((signed short)(-SHRT_MAX - 1))
+#endif
+
+#if !defined(UINT_MAX)
+ #define UINT_MAX ((unsigned int)~0U)
+#endif
+
+#if !defined(INT_MAX)
+ #define INT_MAX ((signed int)(UINT_MAX >> 1))
+#endif
+#if !defined(INT_MIN)
+ #define INT_MIN ((signed int)(-INT_MAX - 1))
+#endif
+
+#if !defined(ULONG_MAX)
+ #define ULONG_MAX ((unsigned long)~0UL)
+#endif
+
+#if !defined(LONG_MAX)
+ #define LONG_MAX ((signed long)(ULONG_MAX >> 1))
+#endif
+#if !defined(LONG_MIN)
+ #define LONG_MIN ((signed long)(-LONG_MAX - 1))
+#endif
+
+#if !defined(ULLONG_MAX)
+ #define ULLONG_MAX ((unsigned long long)~0ULL)
+#endif
+
+#if !defined(LLONG_MAX)
+ #define LLONG_MAX ((signed long long)(ULLONG_MAX >> 1))
+#endif
+#if !defined(LLONG_MIN)
+ #define LLONG_MIN ((signed long long)(-LLONG_MAX - 1))
+#endif
+
+#if !defined(SIZE_MAX)
+ #if 1 == CSTD_CPU_32BIT
+ #define SIZE_MAX UINT32_MAX
+ #elif 1 == CSTD_CPU_64BIT
+ #define SIZE_MAX UINT64_MAX
+ #endif
+#endif
+
+/* ============================================================================
+ Keywords
+============================================================================ */
+/* Portable keywords. */
+
+#if !defined(CONST)
+/**
+ * @hideinitializer
+ * Variable is a C @c const, which can be made non-const for testing purposes.
+ */
+ #define CONST const
+#endif
+
+#if !defined(STATIC)
+/**
+ * @hideinitializer
+ * Variable is a C @c static, which can be made non-static for testing
+ * purposes.
+ */
+ #define STATIC static
+#endif
+
+/**
+ * Specifies a function as being exported outside of a logical module.
+ */
+#define PUBLIC
+
+/**
+ * @def PROTECTED
+ * Specifies a a function which is internal to an logical module, but which
+ * should not be used outside of that module. This cannot be enforced by the
+ * compiler, as a module is typically more than one translation unit.
+ */
+#define PROTECTED
+
+/**
+ * Specifies a function as being internal to a translation unit. Private
+ * functions would typically be declared as STATIC, unless they are being
+ * exported for unit test purposes.
+ */
+#define PRIVATE STATIC
+
+/**
+ * Specify an assertion value which is evaluated at compile time. Recommended
+ * usage is specification of a @c static @c INLINE function containing all of
+ * the assertions thus:
+ *
+ * @code
+ * static INLINE [module]_compile_time_assertions( void )
+ * {
+ * COMPILE_TIME_ASSERT( sizeof(uintptr_t) == sizeof(intptr_t) );
+ * }
+ * @endcode
+ *
+ * @note Use @c static not @c STATIC. We never want to turn off this @c static
+ * specification for testing purposes.
+ */
+#define CSTD_COMPILE_TIME_ASSERT( expr ) \
+ do { switch(0){case 0: case (expr):;} } while( FALSE )
+
+/**
+ * @hideinitializer
+ * @deprecated Prefered form is @c CSTD_UNUSED
+ * Function-like macro for suppressing unused variable warnings. Where possible
+ * such variables should be removed; this macro is present for cases where we
+ * much support API backwards compatibility.
+ */
+#define UNUSED( x ) ((void)(x))
+
+/**
+ * @hideinitializer
+ * Function-like macro for suppressing unused variable warnings. Where possible
+ * such variables should be removed; this macro is present for cases where we
+ * much support API backwards compatibility.
+ */
+#define CSTD_UNUSED( x ) ((void)(x))
+
+/**
+ * @hideinitializer
+ * Function-like macro for use where "no behavior" is desired. This is useful
+ * when compile time macros turn a function-like macro in to a no-op, but
+ * where having no statement is otherwise invalid.
+ */
+#define CSTD_NOP( ... ) ((void)#__VA_ARGS__)
+
+/**
+ * @hideinitializer
+ * Function-like macro for converting a pointer in to a u64 for storing into
+ * an external data structure. This is commonly used when pairing a 32-bit
+ * CPU with a 64-bit peripheral, such as a Midgard GPU. C's type promotion
+ * is complex and a straight cast does not work reliably as pointers are
+ * often considered as signed.
+ */
+#define CSTD_PTR_TO_U64( x ) ((uint64_t)((uintptr_t)(x)))
+
+/**
+ * @hideinitializer
+ * Function-like macro for stringizing a single level macro.
+ * @code
+ * #define MY_MACRO 32
+ * CSTD_STR1( MY_MACRO )
+ * > "MY_MACRO"
+ * @endcode
+ */
+#define CSTD_STR1( x ) #x
+
+/**
+ * @hideinitializer
+ * Function-like macro for stringizing a macro's value. This should not be used
+ * if the macro is defined in a way which may have no value; use the
+ * alternative @c CSTD_STR2N macro should be used instead.
+ * @code
+ * #define MY_MACRO 32
+ * CSTD_STR2( MY_MACRO )
+ * > "32"
+ * @endcode
+ */
+#define CSTD_STR2( x ) CSTD_STR1( x )
+
+/**
+ * @hideinitializer
+ * Utility function for stripping the first character off a string.
+ */
+static INLINE char* arm_cstd_strstrip( char * string )
+{
+ return ++string;
+}
+
+/**
+ * @hideinitializer
+ * Function-like macro for stringizing a single level macro where the macro
+ * itself may not have a value. Parameter @c a should be set to any single
+ * character which is then stripped by the macro via an inline function. This
+ * should only be used via the @c CSTD_STR2N macro; for printing a single
+ * macro only the @c CSTD_STR1 macro is a better alternative.
+ *
+ * This macro requires run-time code to handle the case where the macro has
+ * no value (you can't concat empty strings in the preprocessor).
+ */
+#define CSTD_STR1N( a, x ) arm_cstd_strstrip( CSTD_STR1( a##x ) )
+
+/**
+ * @hideinitializer
+ * Function-like macro for stringizing a two level macro where the macro itself
+ * may not have a value.
+ * @code
+ * #define MY_MACRO 32
+ * CSTD_STR2N( MY_MACRO )
+ * > "32"
+ *
+ * #define MY_MACRO 32
+ * CSTD_STR2N( MY_MACRO )
+ * > "32"
+ * @endcode
+ */
+#define CSTD_STR2N( x ) CSTD_STR1N( _, x )
+
+/* ============================================================================
+ Validate portability constructs
+============================================================================ */
+static INLINE void arm_cstd_compile_time_assertions( void )
+{
+ CSTD_COMPILE_TIME_ASSERT( sizeof(uint8_t) == 1 );
+ CSTD_COMPILE_TIME_ASSERT( sizeof(int8_t) == 1 );
+ CSTD_COMPILE_TIME_ASSERT( sizeof(uint16_t) == 2 );
+ CSTD_COMPILE_TIME_ASSERT( sizeof(int16_t) == 2 );
+ CSTD_COMPILE_TIME_ASSERT( sizeof(uint32_t) == 4 );
+ CSTD_COMPILE_TIME_ASSERT( sizeof(int32_t) == 4 );
+ CSTD_COMPILE_TIME_ASSERT( sizeof(uint64_t) == 8 );
+ CSTD_COMPILE_TIME_ASSERT( sizeof(int64_t) == 8 );
+ CSTD_COMPILE_TIME_ASSERT( sizeof(intptr_t) == sizeof(uintptr_t) );
+
+ CSTD_COMPILE_TIME_ASSERT( 1 == TRUE );
+ CSTD_COMPILE_TIME_ASSERT( 0 == FALSE );
+
+#if 1 == CSTD_CPU_32BIT
+ CSTD_COMPILE_TIME_ASSERT( sizeof(uintptr_t) == 4 );
+#elif 1 == CSTD_CPU_64BIT
+ CSTD_COMPILE_TIME_ASSERT( sizeof(uintptr_t) == 8 );
+#endif
+
+}
+
+/* ============================================================================
+ Useful function-like macro
+============================================================================ */
+/**
+ * @brief Return the lesser of two values.
+ * As a macro it may evaluate its arguments more than once.
+ * @see CSTD_MAX
+ */
+#define CSTD_MIN( x, y ) ((x) < (y) ? (x) : (y))
+
+/**
+ * @brief Return the greater of two values.
+ * As a macro it may evaluate its arguments more than once.
+ * If called on the same two arguments as CSTD_MIN it is guaranteed to return
+ * the one that CSTD_MIN didn't return. This is significant for types where not
+ * all values are comparable e.g. NaNs in floating-point types. But if you want
+ * to retrieve the min and max of two values, consider using a conditional swap
+ * instead.
+ */
+#define CSTD_MAX( x, y ) ((x) < (y) ? (y) : (x))
+
+/**
+ * @brief Clamp value @c x to within @c min and @c max inclusive.
+ */
+#define CSTD_CLAMP( x, min, max ) ((x)<(min) ? (min):((x)>(max) ? (max):(x)))
+
+/**
+ * Flag a cast as a reinterpretation, usually of a pointer type.
+ */
+#define CSTD_REINTERPRET_CAST(type) (type)
+
+/**
+ * Flag a cast as casting away const, usually of a pointer type.
+ */
+#define CSTD_CONST_CAST(type) (type)
+
+/**
+ * Flag a cast as a (potentially complex) value conversion, usually of a
+ * numerical type.
+ */
+#define CSTD_STATIC_CAST(type) (type)
+
+/* ============================================================================
+ Useful bit constants
+============================================================================ */
+/**
+ * @cond arm_cstd_utilities
+ */
+
+/* Common bit constant values, useful in embedded programming. */
+#define F_BIT_0 ((uint32_t)0x00000001)
+#define F_BIT_1 ((uint32_t)0x00000002)
+#define F_BIT_2 ((uint32_t)0x00000004)
+#define F_BIT_3 ((uint32_t)0x00000008)
+#define F_BIT_4 ((uint32_t)0x00000010)
+#define F_BIT_5 ((uint32_t)0x00000020)
+#define F_BIT_6 ((uint32_t)0x00000040)
+#define F_BIT_7 ((uint32_t)0x00000080)
+#define F_BIT_8 ((uint32_t)0x00000100)
+#define F_BIT_9 ((uint32_t)0x00000200)
+#define F_BIT_10 ((uint32_t)0x00000400)
+#define F_BIT_11 ((uint32_t)0x00000800)
+#define F_BIT_12 ((uint32_t)0x00001000)
+#define F_BIT_13 ((uint32_t)0x00002000)
+#define F_BIT_14 ((uint32_t)0x00004000)
+#define F_BIT_15 ((uint32_t)0x00008000)
+#define F_BIT_16 ((uint32_t)0x00010000)
+#define F_BIT_17 ((uint32_t)0x00020000)
+#define F_BIT_18 ((uint32_t)0x00040000)
+#define F_BIT_19 ((uint32_t)0x00080000)
+#define F_BIT_20 ((uint32_t)0x00100000)
+#define F_BIT_21 ((uint32_t)0x00200000)
+#define F_BIT_22 ((uint32_t)0x00400000)
+#define F_BIT_23 ((uint32_t)0x00800000)
+#define F_BIT_24 ((uint32_t)0x01000000)
+#define F_BIT_25 ((uint32_t)0x02000000)
+#define F_BIT_26 ((uint32_t)0x04000000)
+#define F_BIT_27 ((uint32_t)0x08000000)
+#define F_BIT_28 ((uint32_t)0x10000000)
+#define F_BIT_29 ((uint32_t)0x20000000)
+#define F_BIT_30 ((uint32_t)0x40000000)
+#define F_BIT_31 ((uint32_t)0x80000000)
+
+/* Common 2^n size values, useful in embedded programming. */
+#define C_SIZE_1B ((uint32_t)0x00000001)
+#define C_SIZE_2B ((uint32_t)0x00000002)
+#define C_SIZE_4B ((uint32_t)0x00000004)
+#define C_SIZE_8B ((uint32_t)0x00000008)
+#define C_SIZE_16B ((uint32_t)0x00000010)
+#define C_SIZE_32B ((uint32_t)0x00000020)
+#define C_SIZE_64B ((uint32_t)0x00000040)
+#define C_SIZE_128B ((uint32_t)0x00000080)
+#define C_SIZE_256B ((uint32_t)0x00000100)
+#define C_SIZE_512B ((uint32_t)0x00000200)
+#define C_SIZE_1KB ((uint32_t)0x00000400)
+#define C_SIZE_2KB ((uint32_t)0x00000800)
+#define C_SIZE_4KB ((uint32_t)0x00001000)
+#define C_SIZE_8KB ((uint32_t)0x00002000)
+#define C_SIZE_16KB ((uint32_t)0x00004000)
+#define C_SIZE_32KB ((uint32_t)0x00008000)
+#define C_SIZE_64KB ((uint32_t)0x00010000)
+#define C_SIZE_128KB ((uint32_t)0x00020000)
+#define C_SIZE_256KB ((uint32_t)0x00040000)
+#define C_SIZE_512KB ((uint32_t)0x00080000)
+#define C_SIZE_1MB ((uint32_t)0x00100000)
+#define C_SIZE_2MB ((uint32_t)0x00200000)
+#define C_SIZE_4MB ((uint32_t)0x00400000)
+#define C_SIZE_8MB ((uint32_t)0x00800000)
+#define C_SIZE_16MB ((uint32_t)0x01000000)
+#define C_SIZE_32MB ((uint32_t)0x02000000)
+#define C_SIZE_64MB ((uint32_t)0x04000000)
+#define C_SIZE_128MB ((uint32_t)0x08000000)
+#define C_SIZE_256MB ((uint32_t)0x10000000)
+#define C_SIZE_512MB ((uint32_t)0x20000000)
+#define C_SIZE_1GB ((uint32_t)0x40000000)
+#define C_SIZE_2GB ((uint32_t)0x80000000)
+
+/**
+ * @endcond
+ */
+
+/**
+ * @}
+ */
+
+/**
+ * @}
+ */
+
+#endif /* End (_ARM_CSTD_) */
--- /dev/null
+/*
+ * Copyright:
+ * ----------------------------------------------------------------------------
+ * This confidential and proprietary software may be used only as authorized
+ * by a licensing agreement from ARM Limited.
+ * (C) COPYRIGHT 2005-2012 ARM Limited , ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorized copies and
+ * copies may only be made to the extent permitted by a licensing agreement
+ * from ARM Limited.
+ * ----------------------------------------------------------------------------
+ */
+
+#ifndef _ARM_CSTD_COMPILERS_H_
+#define _ARM_CSTD_COMPILERS_H_
+
+/* ============================================================================
+ Document default definitions - assuming nothing set at this point.
+============================================================================ */
+/**
+ * @addtogroup arm_cstd_coding_standard
+ * @{
+ */
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if toolchain is Microsoft Visual Studio, 0
+ * otherwise.
+ */
+#define CSTD_TOOLCHAIN_MSVC 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if toolchain is the GNU Compiler Collection, 0
+ * otherwise.
+ */
+#define CSTD_TOOLCHAIN_GCC 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if toolchain is ARM RealView Compiler Tools, 0
+ * otherwise. Note - if running RVCT in GCC mode this define will be set to 0;
+ * @c CSTD_TOOLCHAIN_GCC and @c CSTD_TOOLCHAIN_RVCT_GCC_MODE will both be
+ * defined as 1.
+ */
+#define CSTD_TOOLCHAIN_RVCT 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if toolchain is ARM RealView Compiler Tools running
+ * in GCC mode, 0 otherwise.
+ */
+#define CSTD_TOOLCHAIN_RVCT_GCC_MODE 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if processor is an x86 32-bit machine, 0 otherwise.
+ */
+#define CSTD_CPU_X86_32 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if processor is an x86-64 (AMD64) machine, 0
+ * otherwise.
+ */
+#define CSTD_CPU_X86_64 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if processor is an ARM machine, 0 otherwise.
+ */
+#define CSTD_CPU_ARM 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if processor is a MIPS machine, 0 otherwise.
+ */
+#define CSTD_CPU_MIPS 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if CPU is 32-bit, 0 otherwise.
+ */
+#define CSTD_CPU_32BIT 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if CPU is 64-bit, 0 otherwise.
+ */
+#define CSTD_CPU_64BIT 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if processor configured as big-endian, 0 if it
+ * is little-endian.
+ */
+#define CSTD_CPU_BIG_ENDIAN 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if operating system is a version of Windows, 0 if
+ * it is not.
+ */
+#define CSTD_OS_WINDOWS 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if operating system is a 32-bit version of Windows,
+ * 0 if it is not.
+ */
+#define CSTD_OS_WIN32 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if operating system is a 64-bit version of Windows,
+ * 0 if it is not.
+ */
+#define CSTD_OS_WIN64 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if operating system is Linux, 0 if it is not.
+ */
+#define CSTD_OS_LINUX 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if we are compiling Linux kernel code, 0 otherwise.
+ */
+#define CSTD_OS_LINUX_KERNEL 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if operating system is a 32-bit version of Linux,
+ * 0 if it is not.
+ */
+#define CSTD_OS_LINUX32 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if operating system is a 64-bit version of Linux,
+ * 0 if it is not.
+ */
+#define CSTD_OS_LINUX64 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if operating system is Android, 0 if it is not.
+ */
+#define CSTD_OS_ANDROID 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if we are compiling Android kernel code, 0 otherwise.
+ */
+#define CSTD_OS_ANDROID_KERNEL 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if operating system is a 32-bit version of Android,
+ * 0 if it is not.
+ */
+#define CSTD_OS_ANDROID32 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if operating system is a 64-bit version of Android,
+ * 0 if it is not.
+ */
+#define CSTD_OS_ANDROID64 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if operating system is a version of Apple OS,
+ * 0 if it is not.
+ */
+#define CSTD_OS_APPLEOS 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if operating system is a 32-bit version of Apple OS,
+ * 0 if it is not.
+ */
+#define CSTD_OS_APPLEOS32 0
+
+/**
+ * @hideinitializer
+ * Defined with value of 1 if operating system is a 64-bit version of Apple OS,
+ * 0 if it is not.
+ */
+#define CSTD_OS_APPLEOS64 0
+
+/**
+ * @def CSTD_OS_SYMBIAN
+ * @hideinitializer
+ * Defined with value of 1 if operating system is Symbian, 0 if it is not.
+ */
+#define CSTD_OS_SYMBIAN 0
+
+/**
+ * @def CSTD_OS_NONE
+ * @hideinitializer
+ * Defined with value of 1 if there is no operating system (bare metal), 0
+ * otherwise
+ */
+#define CSTD_OS_NONE 0
+
+/* ============================================================================
+ Determine the compiler in use
+============================================================================ */
+#if defined(_MSC_VER)
+ #undef CSTD_TOOLCHAIN_MSVC
+ #define CSTD_TOOLCHAIN_MSVC 1
+
+#elif defined(__GNUC__)
+ #undef CSTD_TOOLCHAIN_GCC
+ #define CSTD_TOOLCHAIN_GCC 1
+
+ /* Detect RVCT pretending to be GCC. */
+ #if defined(__ARMCC_VERSION)
+ #undef CSTD_TOOLCHAIN_RVCT_GCC_MODE
+ #define CSTD_TOOLCHAIN_RVCT_GCC_MODE 1
+ #endif
+
+#elif defined(__ARMCC_VERSION)
+ #undef CSTD_TOOLCHAIN_RVCT
+ #define CSTD_TOOLCHAIN_RVCT 1
+
+#else
+ #warning "Unsupported or unknown toolchain"
+
+#endif
+
+/* ============================================================================
+ Determine the processor
+============================================================================ */
+#if 1 == CSTD_TOOLCHAIN_MSVC
+ #if defined(_M_IX86)
+ #undef CSTD_CPU_X86_32
+ #define CSTD_CPU_X86_32 1
+
+ #elif defined(_M_X64) || defined(_M_AMD64)
+ #undef CSTD_CPU_X86_64
+ #define CSTD_CPU_X86_64 1
+
+ #elif defined(_M_ARM)
+ #undef CSTD_CPU_ARM
+ #define CSTD_CPU_ARM 1
+
+ #elif defined(_M_MIPS)
+ #undef CSTD_CPU_MIPS
+ #define CSTD_CPU_MIPS 1
+
+ #else
+ #warning "Unsupported or unknown host CPU for MSVC tools"
+
+ #endif
+
+#elif 1 == CSTD_TOOLCHAIN_GCC
+ #if defined(__amd64__)
+ #undef CSTD_CPU_X86_64
+ #define CSTD_CPU_X86_64 1
+
+ #elif defined(__i386__)
+ #undef CSTD_CPU_X86_32
+ #define CSTD_CPU_X86_32 1
+
+ #elif defined(__arm__)
+ #undef CSTD_CPU_ARM
+ #define CSTD_CPU_ARM 1
+
+ #elif defined(__mips__)
+ #undef CSTD_CPU_MIPS
+ #define CSTD_CPU_MIPS 1
+
+ #else
+ #warning "Unsupported or unknown host CPU for GCC tools"
+
+ #endif
+
+#elif 1 == CSTD_TOOLCHAIN_RVCT
+ #undef CSTD_CPU_ARM
+ #define CSTD_CPU_ARM 1
+
+#else
+ #warning "Unsupported or unknown toolchain"
+
+#endif
+
+/* ============================================================================
+ Determine the Processor Endianness
+============================================================================ */
+
+#if ((1 == CSTD_CPU_X86_32) || (1 == CSTD_CPU_X86_64))
+ /* Note: x86 and x86-64 are always little endian, so leave at default. */
+
+#elif 1 == CSTD_TOOLCHAIN_RVCT
+ #if defined(__BIG_ENDIAN)
+ #undef CSTD_ENDIAN_BIG
+ #define CSTD_ENDIAN_BIG 1
+ #endif
+
+#elif ((1 == CSTD_TOOLCHAIN_GCC) && (1 == CSTD_CPU_ARM))
+ #if defined(__ARMEB__)
+ #undef CSTD_ENDIAN_BIG
+ #define CSTD_ENDIAN_BIG 1
+ #endif
+
+#elif ((1 == CSTD_TOOLCHAIN_GCC) && (1 == CSTD_CPU_MIPS))
+ #if defined(__MIPSEB__)
+ #undef CSTD_ENDIAN_BIG
+ #define CSTD_ENDIAN_BIG 1
+ #endif
+
+#elif 1 == CSTD_TOOLCHAIN_MSVC
+ /* Note: Microsoft only support little endian, so leave at default. */
+
+#else
+ #warning "Unsupported or unknown CPU"
+
+#endif
+
+/* ============================================================================
+ Determine the operating system and addressing width
+============================================================================ */
+#if 1 == CSTD_TOOLCHAIN_MSVC
+ #if defined(_WIN32) && !defined(_WIN64)
+ #undef CSTD_OS_WINDOWS
+ #define CSTD_OS_WINDOWS 1
+ #undef CSTD_OS_WIN32
+ #define CSTD_OS_WIN32 1
+ #undef CSTD_CPU_32BIT
+ #define CSTD_CPU_32BIT 1
+
+ #elif defined(_WIN32) && defined(_WIN64)
+ #undef CSTD_OS_WINDOWS
+ #define CSTD_OS_WINDOWS 1
+ #undef CSTD_OS_WIN64
+ #define CSTD_OS_WIN64 1
+ #undef CSTD_CPU_64BIT
+ #define CSTD_CPU_64BIT 1
+
+ #else
+ #warning "Unsupported or unknown host OS for MSVC tools"
+
+ #endif
+
+#elif 1 == CSTD_TOOLCHAIN_GCC
+ #if defined(_WIN32) && defined(_WIN64)
+ #undef CSTD_OS_WINDOWS
+ #define CSTD_OS_WINDOWS 1
+ #undef CSTD_OS_WIN64
+ #define CSTD_OS_WIN64 1
+ #undef CSTD_CPU_64BIT
+ #define CSTD_CPU_64BIT 1
+
+ #elif defined(_WIN32) && !defined(_WIN64)
+ #undef CSTD_OS_WINDOWS
+ #define CSTD_OS_WINDOWS 1
+ #undef CSTD_OS_WIN32
+ #define CSTD_OS_WIN32 1
+ #undef CSTD_CPU_32BIT
+ #define CSTD_CPU_32BIT 1
+
+ #elif defined(ANDROID)
+ #undef CSTD_OS_ANDROID
+ #define CSTD_OS_ANDROID 1
+
+ #if defined(__KERNEL__)
+ #undef CSTD_OS_ANDROID_KERNEL
+ #define CSTD_OS_ANDROID_KERNEL 1
+ #endif
+
+ #if defined(__LP64__) || defined(_LP64)
+ #undef CSTD_OS_ANDROID64
+ #define CSTD_OS_ANDROID64 1
+ #undef CSTD_CPU_64BIT
+ #define CSTD_CPU_64BIT 1
+ #else
+ #undef CSTD_OS_ANDROID32
+ #define CSTD_OS_ANDROID32 1
+ #undef CSTD_CPU_32BIT
+ #define CSTD_CPU_32BIT 1
+ #endif
+
+ #elif defined(__KERNEL__) || defined(__linux)
+ #undef CSTD_OS_LINUX
+ #define CSTD_OS_LINUX 1
+
+ #if defined(__KERNEL__)
+ #undef CSTD_OS_LINUX_KERNEL
+ #define CSTD_OS_LINUX_KERNEL 1
+ #endif
+
+ #if defined(__LP64__) || defined(_LP64)
+ #undef CSTD_OS_LINUX64
+ #define CSTD_OS_LINUX64 1
+ #undef CSTD_CPU_64BIT
+ #define CSTD_CPU_64BIT 1
+ #else
+ #undef CSTD_OS_LINUX32
+ #define CSTD_OS_LINUX32 1
+ #undef CSTD_CPU_32BIT
+ #define CSTD_CPU_32BIT 1
+ #endif
+
+ #elif defined(__APPLE__)
+ #undef CSTD_OS_APPLEOS
+ #define CSTD_OS_APPLEOS 1
+
+ #if defined(__LP64__) || defined(_LP64)
+ #undef CSTD_OS_APPLEOS64
+ #define CSTD_OS_APPLEOS64 1
+ #undef CSTD_CPU_64BIT
+ #define CSTD_CPU_64BIT 1
+ #else
+ #undef CSTD_OS_APPLEOS32
+ #define CSTD_OS_APPLEOS32 1
+ #undef CSTD_CPU_32BIT
+ #define CSTD_CPU_32BIT 1
+ #endif
+
+ #elif defined(__SYMBIAN32__)
+ #undef CSTD_OS_SYMBIAN
+ #define CSTD_OS_SYMBIAN 1
+ #undef CSTD_CPU_32BIT
+ #define CSTD_CPU_32BIT 1
+
+ #else
+ #undef CSTD_OS_NONE
+ #define CSTD_OS_NONE 1
+ #undef CSTD_CPU_32BIT
+ #define CSTD_CPU_32BIT 1
+
+#endif
+
+#elif 1 == CSTD_TOOLCHAIN_RVCT
+
+ #if defined(ANDROID)
+ #undef CSTD_OS_ANDROID
+ #undef CSTD_OS_ANDROID32
+ #define CSTD_OS_ANDROID 1
+ #define CSTD_OS_ANDROID32 1
+
+ #elif defined(__linux)
+ #undef CSTD_OS_LINUX
+ #undef CSTD_OS_LINUX32
+ #define CSTD_OS_LINUX 1
+ #define CSTD_OS_LINUX32 1
+
+ #elif defined(__SYMBIAN32__)
+ #undef CSTD_OS_SYMBIAN
+ #define CSTD_OS_SYMBIAN 1
+
+ #else
+ #undef CSTD_OS_NONE
+ #define CSTD_OS_NONE 1
+
+#endif
+
+#else
+ #warning "Unsupported or unknown host OS"
+
+#endif
+
+/* ============================================================================
+ Determine the correct linker symbol Import and Export Macros
+============================================================================ */
+/**
+ * @defgroup arm_cstd_linkage_specifiers Linkage Specifiers
+ * @{
+ *
+ * This set of macros contain system-dependent linkage specifiers which
+ * determine the visibility of symbols across DLL boundaries. A header for a
+ * particular DLL should define a set of local macros derived from these,
+ * and should not use these macros to decorate functions directly as there may
+ * be multiple DLLs being used.
+ *
+ * These DLL library local macros should be (with appropriate library prefix)
+ * <tt>[MY_LIBRARY]_API</tt>, <tt>[MY_LIBRARY]_IMPL</tt>, and
+ * <tt>[MY_LIBRARY]_LOCAL</tt>.
+ *
+ * - <tt>[MY_LIBRARY]_API</tt> should be use to decorate the function
+ * declarations in the header. It should be defined as either
+ * @c CSTD_LINK_IMPORT or @c CSTD_LINK_EXPORT, depending whether the
+ * current situation is a compile of the DLL itself (use export) or a
+ * compile of an external user of the DLL (use import).
+ * - <tt>[MY_LIBRARY]_IMPL</tt> should be defined as @c CSTD_LINK_IMPL
+ * and should be used to decorate the definition of functions in the C
+ * file.
+ * - <tt>[MY_LIBRARY]_LOCAL</tt> should be used to decorate function
+ * declarations which are exported across translation units within the
+ * DLL, but which are not exported outside of the DLL boundary.
+ *
+ * Functions which are @c static in either a C file or in a header file do not
+ * need any form of linkage decoration, and should therefore have no linkage
+ * macro applied to them.
+ */
+
+/**
+ * @def CSTD_LINK_IMPORT
+ * Specifies a function as being imported to a translation unit across a DLL
+ * boundary.
+ */
+
+/**
+ * @def CSTD_LINK_EXPORT
+ * Specifies a function as being exported across a DLL boundary by a
+ * translation unit.
+ */
+
+/**
+ * @def CSTD_LINK_IMPL
+ * Specifies a function which will be exported across a DLL boundary as
+ * being implemented by a translation unit.
+ */
+
+/**
+ * @def CSTD_LINK_LOCAL
+ * Specifies a function which is internal to a DLL, and which should not be
+ * exported outside of it.
+ */
+
+/**
+ * @}
+ */
+
+#if 1 == CSTD_OS_LINUX
+ #define CSTD_LINK_IMPORT __attribute__((visibility("default")))
+ #define CSTD_LINK_EXPORT __attribute__((visibility("default")))
+ #define CSTD_LINK_IMPL __attribute__((visibility("default")))
+ #define CSTD_LINK_LOCAL __attribute__((visibility("hidden")))
+
+#elif 1 == CSTD_OS_WINDOWS
+ #define CSTD_LINK_IMPORT __declspec(dllimport)
+ #define CSTD_LINK_EXPORT __declspec(dllexport)
+ #define CSTD_LINK_IMPL __declspec(dllexport)
+ #define CSTD_LINK_LOCAL
+
+#elif 1 == CSTD_OS_SYMBIAN
+ #define CSTD_LINK_IMPORT IMPORT_C
+ #define CSTD_LINK_EXPORT IMPORT_C
+ #define CSTD_LINK_IMPL EXPORT_C
+ #define CSTD_LINK_LOCAL
+
+#elif 1 == CSTD_OS_APPLEOS
+ #define CSTD_LINK_IMPORT __attribute__((visibility("default")))
+ #define CSTD_LINK_EXPORT __attribute__((visibility("default")))
+ #define CSTD_LINK_IMPL __attribute__((visibility("default")))
+ #define CSTD_LINK_LOCAL __attribute__((visibility("hidden")))
+
+#else /* CSTD_OS_NONE */
+ #define CSTD_LINK_IMPORT
+ #define CSTD_LINK_EXPORT
+ #define CSTD_LINK_IMPL
+ #define CSTD_LINK_LOCAL
+
+#endif
+
+/**
+ * @}
+ */
+
+#endif /* End (_ARM_CSTD_COMPILERS_H_) */
--- /dev/null
+/*
+ * Copyright:
+ * ----------------------------------------------------------------------------
+ * This confidential and proprietary software may be used only as authorized
+ * by a licensing agreement from ARM Limited.
+ * (C) COPYRIGHT 2009-2010 ARM Limited , ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorized copies and
+ * copies may only be made to the extent permitted by a licensing agreement
+ * from ARM Limited.
+ * ----------------------------------------------------------------------------
+ */
+
+#ifndef _ARM_CSTD_PACK_POP_H_
+#define _ARM_CSTD_PACK_POP_H_
+
+#if 1 == CSTD_TOOLCHAIN_MSVC
+ #include <poppack.h>
+#endif
+
+#endif /* End (_ARM_CSTD_PACK_POP_H_) */
--- /dev/null
+/*
+ * Copyright:
+ * ----------------------------------------------------------------------------
+ * This confidential and proprietary software may be used only as authorized
+ * by a licensing agreement from ARM Limited.
+ * (C) COPYRIGHT 2009-2010 ARM Limited , ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorized copies and
+ * copies may only be made to the extent permitted by a licensing agreement
+ * from ARM Limited.
+ * ----------------------------------------------------------------------------
+ */
+
+#ifndef _ARM_CSTD_PACK_PUSH_H_
+#define _ARM_CSTD_PACK_PUSH_H_
+
+#if 1 == CSTD_TOOLCHAIN_MSVC
+ #include <pshpack1.h>
+#endif
+
+#endif /* End (_ARM_CSTD_PACK_PUSH_H_) */
--- /dev/null
+/*
+ * Copyright:
+ * ----------------------------------------------------------------------------
+ * This confidential and proprietary software may be used only as authorized
+ * by a licensing agreement from ARM Limited.
+ * (C) COPYRIGHT 2009-2010 ARM Limited , ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorized copies and
+ * copies may only be made to the extent permitted by a licensing agreement
+ * from ARM Limited.
+ * ----------------------------------------------------------------------------
+ */
+
+#ifndef _ARM_CSTD_TYPES_H_
+#define _ARM_CSTD_TYPES_H_
+
+#if 1 == CSTD_TOOLCHAIN_MSVC
+ #include "arm_cstd_types_msvc.h"
+#elif 1 == CSTD_TOOLCHAIN_GCC
+ #include "arm_cstd_types_gcc.h"
+#elif 1 == CSTD_TOOLCHAIN_RVCT
+ #include "arm_cstd_types_rvct.h"
+#else
+ #error "Toolchain not recognized"
+#endif
+
+#endif /* End (_ARM_CSTD_TYPES_H_) */
--- /dev/null
+/*
+ * Copyright:
+ * ----------------------------------------------------------------------------
+ * This confidential and proprietary software may be used only as authorized
+ * by a licensing agreement from ARM Limited.
+ * (C) COPYRIGHT 2009-2011 ARM Limited , ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorized copies and
+ * copies may only be made to the extent permitted by a licensing agreement
+ * from ARM Limited.
+ * ----------------------------------------------------------------------------
+ */
+
+#ifndef _ARM_CSTD_TYPES_GCC_H_
+#define _ARM_CSTD_TYPES_GCC_H_
+
+/* ============================================================================
+ Type definitions
+============================================================================ */
+/* All modern versions of GCC support stdint outside of C99 Mode. */
+/* However, Linux kernel limits what headers are available! */
+#if 1 == CSTD_OS_LINUX_KERNEL
+ #include <linux/kernel.h>
+ #include <linux/types.h>
+ #include <linux/stddef.h>
+ #include <linux/version.h>
+
+ /* Fix up any types which CSTD provdes but which Linux is missing. */
+ /* Note Linux assumes pointers are "long", so this is safe. */
+ #if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,24)
+ typedef unsigned long uintptr_t;
+ #endif
+ typedef long intptr_t;
+
+#else
+ #include <stdint.h>
+ #include <stddef.h>
+ #include <limits.h>
+#endif
+
+typedef uint32_t bool_t;
+
+#if !defined(TRUE)
+ #define TRUE ((bool_t)1)
+#endif
+
+#if !defined(FALSE)
+ #define FALSE ((bool_t)0)
+#endif
+
+/* ============================================================================
+ Keywords
+============================================================================ */
+/* Doxygen documentation for these is in the RVCT header. */
+#define ASM __asm__
+
+#define INLINE __inline__
+
+#define FORCE_INLINE __attribute__((__always_inline__)) __inline__
+
+#define NEVER_INLINE __attribute__((__noinline__))
+
+#define PURE __attribute__((__pure__))
+
+#define PACKED __attribute__((__packed__))
+
+/* GCC does not support pointers to UNALIGNED data, so we do not define it to
+ * force a compile error if this macro is used. */
+
+#define RESTRICT __restrict__
+
+/* RVCT in GCC mode does not support the CHECK_RESULT attribute. */
+#if 0 == CSTD_TOOLCHAIN_RVCT_GCC_MODE
+ #define CHECK_RESULT __attribute__((__warn_unused_result__))
+#else
+ #define CHECK_RESULT
+#endif
+
+/* RVCT in GCC mode does not support the __func__ name outside of C99. */
+#if (0 == CSTD_TOOLCHAIN_RVCT_GCC_MODE)
+ #define CSTD_FUNC __func__
+#else
+ #define CSTD_FUNC __FUNCTION__
+#endif
+
+#endif /* End (_ARM_CSTD_TYPES_GCC_H_) */
--- /dev/null
+/*
+ * Copyright:
+ * ----------------------------------------------------------------------------
+ * This confidential and proprietary software may be used only as authorized
+ * by a licensing agreement from ARM Limited.
+ * (C) COPYRIGHT 2009-2011 ARM Limited , ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorized copies and
+ * copies may only be made to the extent permitted by a licensing agreement
+ * from ARM Limited.
+ * ----------------------------------------------------------------------------
+ */
+
+#ifndef _ARM_CSTD_TYPES_RVCT_H_
+#define _ARM_CSTD_TYPES_RVCT_H_
+
+/* ============================================================================
+ Type definitions
+============================================================================ */
+#include <stddef.h>
+#include <limits.h>
+
+#if 199901L <= __STDC_VERSION__
+ #include <inttypes.h>
+#else
+ typedef unsigned char uint8_t;
+ typedef signed char int8_t;
+ typedef unsigned short uint16_t;
+ typedef signed short int16_t;
+ typedef unsigned int uint32_t;
+ typedef signed int int32_t;
+ typedef unsigned __int64 uint64_t;
+ typedef signed __int64 int64_t;
+ typedef ptrdiff_t intptr_t;
+ typedef size_t uintptr_t;
+#endif
+
+typedef uint32_t bool_t;
+
+#if !defined(TRUE)
+ #define TRUE ((bool_t)1)
+#endif
+
+#if !defined(FALSE)
+ #define FALSE ((bool_t)0)
+#endif
+
+/* ============================================================================
+ Keywords
+============================================================================ */
+/**
+ * @addtogroup arm_cstd_coding_standard
+ * @{
+ */
+
+/**
+ * @def ASM
+ * @hideinitializer
+ * Mark an assembler block. Such blocks are often compiler specific, so often
+ * need to be surrounded in appropriate @c ifdef and @c endif blocks
+ * using the relevant @c CSTD_TOOLCHAIN macro.
+ */
+#define ASM __asm
+
+/**
+ * @def INLINE
+ * @hideinitializer
+ * Mark a definition as something which should be inlined. This is not always
+ * possible on a given compiler, and may be disabled at lower optimization
+ * levels.
+ */
+#define INLINE __inline
+
+/**
+ * @def FORCE_INLINE
+ * @hideinitializer
+ * Mark a definition as something which should be inlined. This provides a much
+ * stronger hint to the compiler than @c INLINE, and if supported should always
+ * result in an inlined function being emitted. If not supported this falls
+ * back to using the @c INLINE definition.
+ */
+#define FORCE_INLINE __forceinline
+
+/**
+ * @def NEVER_INLINE
+ * @hideinitializer
+ * Mark a definition as something which should not be inlined. This provides a
+ * stronger hint to the compiler than the function should not be inlined,
+ * bypassing any heuristic rules the compiler normally applies. If not
+ * supported by a toolchain this falls back to being an empty macro.
+ */
+#define NEVER_INLINE __declspec(noinline)
+
+/**
+ * @def PURE
+ * @hideinitializer
+ * Denotes that a function's return is only dependent on its inputs, enabling
+ * more efficient optimizations. Falls back to an empty macro if not supported.
+ */
+#define PURE __pure
+
+/**
+ * @def PACKED
+ * @hideinitializer
+ * Denotes that a structure should be stored in a packed form. This macro must
+ * be used in conjunction with the @c arm_cstd_pack_* headers for portability:
+ *
+ * @code
+ * #include <cstd/arm_cstd_pack_push.h>
+ *
+ * struct PACKED myStruct {
+ * ...
+ * };
+ *
+ * #include <cstd/arm_cstd_pack_pop.h>
+ * PACKED
+ * @endcode
+ */
+#define PACKED __packed
+
+/**
+ * @def UNALIGNED
+ * @hideinitializer
+ * Denotes that a pointer points to a buffer with lower alignment than the
+ * natural alignment required by the C standard. This should only be used
+ * in extreme cases, as the emitted code is normally more efficient if memory
+ * is aligned.
+ *
+ * @warning This is \b NON-PORTABLE. The GNU tools are anti-unaligned pointers
+ * and have no support for such a construction.
+ */
+#define UNALIGNED __packed
+
+/**
+ * @def RESTRICT
+ * @hideinitializer
+ * Denotes that a pointer does not overlap with any other points currently in
+ * scope, increasing the range of optimizations which can be performed by the
+ * compiler.
+ *
+ * @warning Specification of @c RESTRICT is a contract between the programmer
+ * and the compiler. If you place @c RESTICT on buffers which do actually
+ * overlap the behavior is undefined, and likely to vary at different
+ * optimization levels.!
+ */
+#define RESTRICT __restrict
+
+/**
+ * @def CHECK_RESULT
+ * @hideinitializer
+ * Function attribute which causes a warning to be emitted if the compiler's
+ * return value is not used by the caller. Compiles to an empty macro if
+ * there is no supported mechanism for this check in the underlying compiler.
+ *
+ * @note At the time of writing this is only supported by GCC. RVCT does not
+ * support this attribute, even in GCC mode, so engineers are encouraged to
+ * compile their code using GCC even if primarily working with another
+ * compiler.
+ *
+ * @code
+ * CHECK_RESULT int my_func( void );
+ * @endcode
+ */
+#define CHECK_RESULT
+
+/**
+ * @def CSTD_FUNC
+ * Specify the @c CSTD_FUNC macro, a portable construct containing the name of
+ * the current function. On most compilers it is illegal to use this macro
+ * outside of a function scope. If not supported by the compiler we define
+ * @c CSTD_FUNC as an empty string.
+ *
+ * @warning Due to the implementation of this on most modern compilers this
+ * expands to a magically defined "static const" variable, not a constant
+ * string. This makes injecting @c CSTD_FUNC directly in to compile-time
+ * strings impossible, so if you want to make the function name part of a
+ * larger string you must use a printf-like function with a @c @%s template
+ * which is populated with @c CSTD_FUNC
+ */
+#define CSTD_FUNC __FUNCTION__
+
+/**
+ * @}
+ */
+
+#endif /* End (_ARM_CSTD_TYPES_RVCT_H_) */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _MALISW_H_
+#define _MALISW_H_
+
+#define MALI_MODULE_MALISW_MAJOR 2
+#define MALI_MODULE_MALISW_MINOR 4
+
+/**
+ * @file mali_malisw.h
+ * Driver-wide include for common macros and types.
+ */
+
+/**
+ * @defgroup malisw Mali software definitions and types
+ * @{
+ */
+
+#include <stddef.h>
+
+#include "mali_stdtypes.h"
+#include "mali_version_macros.h"
+
+/** @brief Gets the container object when given a pointer to a member of an object. */
+#define CONTAINER_OF(ptr, type, member) ((type *)((char *)(ptr) - offsetof(type,member)))
+
+/** @brief Gets the number of elements of type s in a fixed length array of s */
+#define NELEMS(s) (sizeof(s)/sizeof((s)[0]))
+
+/**
+ * @brief The lesser of two values.
+ * May evaluate its arguments more than once.
+ * @see CSTD_MIN
+ */
+#define MIN(x,y) CSTD_MIN(x,y)
+
+/**
+ * @brief The greater of two values.
+ * May evaluate its arguments more than once.
+ * @see CSTD_MAX
+ */
+#define MAX(x,y) CSTD_MAX(x,y)
+
+/**
+ * @brief Clamp value x to within min and max inclusive
+ * May evaluate its arguments more than once.
+ * @see CSTD_CLAMP
+ */
+#define CLAMP( x, min, max ) CSTD_CLAMP( x, min, max )
+
+/**
+ * @brief Convert a pointer into a u64 for storing in a data structure.
+ * This is commonly used when pairing a 32-bit CPU with a 64-bit peripheral,
+ * such as a Midgard GPU. C's type promotion is complex and a straight cast
+ * does not work reliably as pointers are often considered as signed.
+ */
+#define PTR_TO_U64( x ) CSTD_PTR_TO_U64( x )
+
+/**
+ * @name Mali library linkage specifiers
+ * These directly map to the cstd versions described in detail here: @ref arm_cstd_linkage_specifiers
+ * @{
+ */
+#define MALI_IMPORT CSTD_LINK_IMPORT
+#define MALI_EXPORT CSTD_LINK_EXPORT
+#define MALI_IMPL CSTD_LINK_IMPL
+#define MALI_LOCAL CSTD_LINK_LOCAL
+
+/** @brief Decorate exported function prototypes.
+ *
+ * The file containing the implementation of the function should define this to be MALI_EXPORT before including
+ * malisw/mali_malisw.h.
+ */
+#ifndef MALI_API
+#define MALI_API MALI_IMPORT
+#endif
+/** @} */
+
+/** @name Testable static functions
+ * @{
+ *
+ * These macros can be used to allow functions to be static in release builds but exported from a shared library in unit
+ * test builds, allowing them to be tested or used to assist testing.
+ *
+ * Example mali_foo_bar.c containing the function to test:
+ *
+ * @code
+ * #define MALI_API MALI_EXPORT
+ *
+ * #include <malisw/mali_malisw.h>
+ * #include "mali_foo_testable_statics.h"
+ *
+ * MALI_TESTABLE_STATIC_IMPL void my_func()
+ * {
+ * //Implementation
+ * }
+ * @endcode
+ *
+ * Example mali_foo_testable_statics.h:
+ *
+ * @code
+ * #if 1 == MALI_UNIT_TEST
+ * #include <malisw/mali_malisw.h>
+ *
+ * MALI_TESTABLE_STATIC_API void my_func();
+ *
+ * #endif
+ * @endcode
+ *
+ * Example mali_foo_tests.c:
+ *
+ * @code
+ * #include <foo/src/mali_foo_testable_statics.h>
+ *
+ * void my_test_func()
+ * {
+ * my_func();
+ * }
+ * @endcode
+ */
+
+/** @brief Decorate testable static function implementations.
+ *
+ * A header file containing a MALI_TESTABLE_STATIC_API-decorated prototype for each static function will be required
+ * when MALI_UNIT_TEST == 1 in order to link the function from the test.
+ */
+#if 1 == MALI_UNIT_TEST
+#define MALI_TESTABLE_STATIC_IMPL MALI_IMPL
+#else
+#define MALI_TESTABLE_STATIC_IMPL static
+#endif
+
+/** @brief Decorate testable static function prototypes.
+ *
+ * @note Prototypes should @em only be declared when MALI_UNIT_TEST == 1
+ */
+#define MALI_TESTABLE_STATIC_API MALI_API
+/** @} */
+
+/** @name Testable local functions
+ * @{
+ *
+ * These macros can be used to allow functions to be local to a shared library in release builds but be exported in unit
+ * test builds, allowing them to be tested or used to assist testing.
+ *
+ * Example mali_foo_bar.c containing the function to test:
+ *
+ * @code
+ * #define MALI_API MALI_EXPORT
+ *
+ * #include <malisw/mali_malisw.h>
+ * #include "mali_foo_bar.h"
+ *
+ * MALI_TESTABLE_LOCAL_IMPL void my_func()
+ * {
+ * //Implementation
+ * }
+ * @endcode
+ *
+ * Example mali_foo_bar.h:
+ *
+ * @code
+ * #include <malisw/mali_malisw.h>
+ *
+ * MALI_TESTABLE_LOCAL_API void my_func();
+ *
+ * @endcode
+ *
+ * Example mali_foo_tests.c:
+ *
+ * @code
+ * #include <foo/src/mali_foo_bar.h>
+ *
+ * void my_test_func()
+ * {
+ * my_func();
+ * }
+ * @endcode
+ */
+
+/** @brief Decorate testable local function implementations.
+ *
+ * This can be used to have a function normally local to the shared library except in unit test builds where it will be
+ * exported.
+ */
+#if 1 == MALI_UNIT_TEST
+#define MALI_TESTABLE_LOCAL_IMPL MALI_IMPL
+#else
+#define MALI_TESTABLE_LOCAL_IMPL MALI_LOCAL
+#endif
+
+/** @brief Decorate testable local function prototypes.
+ *
+ * This can be used to have a function normally local to the shared library except in unit test builds where it will be
+ * exported.
+ */
+#if 1 == MALI_UNIT_TEST
+#define MALI_TESTABLE_LOCAL_API MALI_API
+#else
+#define MALI_TESTABLE_LOCAL_API MALI_LOCAL
+#endif
+/** @} */
+
+/**
+ * Flag a cast as a reinterpretation, usually of a pointer type.
+ * @see CSTD_REINTERPRET_CAST
+ */
+#define REINTERPRET_CAST(type) CSTD_REINTERPRET_CAST(type)
+
+/**
+ * Flag a cast as casting away const, usually of a pointer type.
+ * @see CSTD_CONST_CAST
+ */
+#define CONST_CAST(type) (type) CSTD_CONST_CAST(type)
+
+/**
+ * Flag a cast as a (potentially complex) value conversion, usually of a numerical type.
+ * @see CSTD_STATIC_CAST
+ */
+#define STATIC_CAST(type) (type) CSTD_STATIC_CAST(type)
+
+
+/** @} */
+
+#endif /* _MALISW_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _MALISW_STDTYPES_H_
+#define _MALISW_STDTYPES_H_
+
+/**
+ * @file mali_stdtypes.h
+ * This file defines the standard types used by the Mali codebase.
+ */
+
+/**
+ * @addtogroup malisw
+ * @{
+ */
+
+/**
+ * @defgroup malisw_stdtypes Mali software standard types
+ *
+ * Basic driver-wide types.
+ */
+
+/**
+ * @addtogroup malisw_stdtypes
+ * @{
+ */
+
+#include "arm_cstd/arm_cstd.h"
+
+/**
+ * @name Scalar types.
+ * These are the scalar types used within the mali driver.
+ * @{
+ */
+/* Note: if compiling the Linux kernel then avoid redefining these. */
+#if 0 == CSTD_OS_LINUX_KERNEL
+ typedef uint64_t u64;
+ typedef uint32_t u32;
+ typedef uint16_t u16;
+ typedef uint8_t u8;
+
+ typedef int64_t s64;
+ typedef int32_t s32;
+ typedef int16_t s16;
+ typedef int8_t s8;
+#endif
+
+typedef double f64;
+typedef float f32;
+typedef u16 f16;
+
+typedef u32 mali_fixed16_16;
+/* @} */
+
+/**
+ * @name Boolean types.
+ * The intended use is for bool8 to be used when storing boolean values in
+ * structures, casting to mali_bool to be used in code sections.
+ * @{
+ */
+typedef bool_t mali_bool;
+typedef u8 mali_bool8;
+
+#define MALI_FALSE FALSE
+#define MALI_TRUE TRUE
+/* @} */
+
+/**
+ * @name Integer bounding values
+ * Maximum and minimum values for integer types
+ * @{
+ */
+#define U64_MAX UINT64_MAX
+#define U32_MAX UINT32_MAX
+#define U16_MAX UINT16_MAX
+#define U8_MAX UINT8_MAX
+
+#define S64_MAX INT64_MAX
+#define S64_MIN INT64_MIN
+#define S32_MAX INT32_MAX
+#define S32_MIN INT32_MIN
+#define S16_MAX INT16_MAX
+#define S16_MIN INT16_MIN
+#define S8_MAX INT8_MAX
+#define S8_MIN INT8_MIN
+/* @} */
+
+/**
+ * @name GPU address types
+ * Types for integers which hold a GPU pointer or GPU pointer offsets.
+ * @{
+ */
+typedef u64 mali_addr64;
+typedef u32 mali_addr32;
+typedef u64 mali_size64;
+typedef s64 mali_offset64;
+/* 32 bit offsets and sizes are always for native types and so use ptrdiff_t and size_t respectively */
+/* @} */
+
+/**
+ * @name Mali error types
+ * @brief The common error type for the mali drivers
+ * The mali_error type, all driver error handling should be of this type unless
+ * it must deal with a specific APIs error type.
+ * @{
+ */
+typedef enum
+{
+ /**
+ * @brief Common Mali errors for the entire driver
+ * MALI_ERROR_NONE is guaranteed to be 0.
+ * @{
+ */
+ MALI_ERROR_NONE = 0,
+ MALI_ERROR_OUT_OF_GPU_MEMORY,
+ MALI_ERROR_OUT_OF_MEMORY,
+ MALI_ERROR_FUNCTION_FAILED,
+ /* @} */
+ /**
+ * @brief Mali errors for Client APIs to pass to EGL when creating EGLImages
+ * These errors must only be returned to EGL from one of the Client APIs as part of the
+ * (clientapi)_egl_image_interface.h
+ * @{
+ */
+ MALI_ERROR_EGLP_BAD_ACCESS,
+ MALI_ERROR_EGLP_BAD_PARAMETER,
+ /* @} */
+ /**
+ * @brief Mali errors for the MCL module.
+ * These errors must only be used within the private components of the OpenCL implementation that report
+ * directly to API functions for cases where errors cannot be detected in the entrypoints file. They must
+ * not be passed between driver components.
+ * These are errors in the mali error space specifically for the MCL module, hence the MCLP prefix.
+ * @{
+ */
+ MALI_ERROR_MCLP_DEVICE_NOT_FOUND,
+ MALI_ERROR_MCLP_DEVICE_NOT_AVAILABLE,
+ MALI_ERROR_MCLP_COMPILER_NOT_AVAILABLE,
+ MALI_ERROR_MCLP_MEM_OBJECT_ALLOCATION_FAILURE,
+ MALI_ERROR_MCLP_PROFILING_INFO_NOT_AVAILABLE,
+ MALI_ERROR_MCLP_MEM_COPY_OVERLAP,
+ MALI_ERROR_MCLP_IMAGE_FORMAT_MISMATCH,
+ MALI_ERROR_MCLP_IMAGE_FORMAT_NOT_SUPPORTED,
+ MALI_ERROR_MCLP_BUILD_PROGRAM_FAILURE,
+ MALI_ERROR_MCLP_MAP_FAILURE,
+ MALI_ERROR_MCLP_MISALIGNED_SUB_BUFFER_OFFSET,
+ MALI_ERROR_MCLP_EXEC_STATUS_ERROR_FOR_EVENTS_IN_WAIT_LIST,
+ MALI_ERROR_MCLP_INVALID_VALUE,
+ MALI_ERROR_MCLP_INVALID_DEVICE_TYPE,
+ MALI_ERROR_MCLP_INVALID_PLATFORM,
+ MALI_ERROR_MCLP_INVALID_DEVICE,
+ MALI_ERROR_MCLP_INVALID_CONTEXT,
+ MALI_ERROR_MCLP_INVALID_QUEUE_PROPERTIES,
+ MALI_ERROR_MCLP_INVALID_COMMAND_QUEUE,
+ MALI_ERROR_MCLP_INVALID_HOST_PTR,
+ MALI_ERROR_MCLP_INVALID_MEM_OBJECT,
+ MALI_ERROR_MCLP_INVALID_IMAGE_FORMAT_DESCRIPTOR,
+ MALI_ERROR_MCLP_INVALID_IMAGE_SIZE,
+ MALI_ERROR_MCLP_INVALID_SAMPLER,
+ MALI_ERROR_MCLP_INVALID_BINARY,
+ MALI_ERROR_MCLP_INVALID_BUILD_OPTIONS,
+ MALI_ERROR_MCLP_INVALID_PROGRAM,
+ MALI_ERROR_MCLP_INVALID_PROGRAM_EXECUTABLE,
+ MALI_ERROR_MCLP_INVALID_KERNEL_NAME,
+ MALI_ERROR_MCLP_INVALID_KERNEL_DEFINITION,
+ MALI_ERROR_MCLP_INVALID_KERNEL,
+ MALI_ERROR_MCLP_INVALID_ARG_INDEX,
+ MALI_ERROR_MCLP_INVALID_ARG_VALUE,
+ MALI_ERROR_MCLP_INVALID_ARG_SIZE,
+ MALI_ERROR_MCLP_INVALID_KERNEL_ARGS,
+ MALI_ERROR_MCLP_INVALID_WORK_DIMENSION,
+ MALI_ERROR_MCLP_INVALID_WORK_GROUP_SIZE,
+ MALI_ERROR_MCLP_INVALID_WORK_ITEM_SIZE,
+ MALI_ERROR_MCLP_INVALID_GLOBAL_OFFSET,
+ MALI_ERROR_MCLP_INVALID_EVENT_WAIT_LIST,
+ MALI_ERROR_MCLP_INVALID_EVENT,
+ MALI_ERROR_MCLP_INVALID_OPERATION,
+ MALI_ERROR_MCLP_INVALID_GL_OBJECT,
+ MALI_ERROR_MCLP_INVALID_BUFFER_SIZE,
+ MALI_ERROR_MCLP_INVALID_MIP_LEVEL,
+ MALI_ERROR_MCLP_INVALID_GLOBAL_WORK_SIZE,
+ /* @} */
+ /**
+ * @brief Mali errors for the BASE module
+ * These errors must only be used within the private components of the Base implementation. They will not
+ * passed to other modules by the base driver.
+ * These are errors in the mali error space specifically for the BASE module, hence the BASEP prefix.
+ * @{
+ */
+ MALI_ERROR_BASEP_INVALID_FUNCTION
+ /* @} */
+} mali_error;
+/* @} */
+
+/* @} */
+
+/* @} */
+
+#endif /* _MALISW_STDTYPES_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _MALISW_VERSION_MACROS_H_
+#define _MALISW_VERSION_MACROS_H_
+
+/**
+ * @file mali_version_macros.h
+ * Mali version control macros.
+ */
+
+/**
+ * @addtogroup malisw
+ * @{
+ */
+
+/**
+ * @defgroup malisw_version Mali module version control
+ *
+ * This file provides a set of macros used to check a module's version. This
+ * version information can be used to perform compile time checks of a module's
+ * suitability for use with another.
+ *
+ * Module versions have both a Major and Minor value which specify the version
+ * of the interface only. These are defined in the following way:
+ *
+ * @li Major: This version is incremented whenever a compatibility-breaking
+ * change is made. For example, removing an interface function.
+ * @li Minor: This version is incremented whenever an interface change is made
+ * that does not break compatibility. For example, adding a new function to the
+ * interface. This value is reset to zero whenever the major number is
+ * incremented.
+ *
+ * When providing a driver module that will be used with this system, the public
+ * header must include a major and minor define of the following form:
+ *
+ * @code
+ * #define MALI_MODULE_<module>_MAJOR X
+ * #define MALI_MODULE_<module>_MINOR Y
+ * @endcode
+ * e.g. for a module CZAM with include czam/mali_czam.h
+ * @code
+ *
+ * #define MALI_MODULE_CZAM_MAJOR 1
+ * #define MALI_MODULE_CZAM_MINOR 0
+ * @endcode
+ *
+ * The version assertion macros outlined below are wrapped with a static function.
+ * This provides more useful error messages when the assertions fail, and allows
+ * the assertions to be specified adjacent to the inclusion of the module header.
+ *
+ * These macros should be used in the global scope of the file. Normal use would be:
+ *
+ * @code
+ * #include <modulex/mali_modulex.h>
+ * #include <moduley/mali_moduley.h>
+ * #include <modulez/mali_modulez.h>
+ * #include <modulez/mali_modulew.h>
+ *
+ * // This module added an enum we needed on minor 4 of major release 2
+ * MALI_MODULE_ASSERT_MAJOR_EQUALS_MINOR_AT_LEAST( MODULEW, 2, 4 )
+ *
+ * // this module really needs to be a specific version
+ * MALI_MODULE_ASSERT_EQUALS( MODULEX, 2, 0 )
+ *
+ * // 1.4 has performance problems
+ * MALI_MODULE_ASSERT_MAXIMUM( MODULEY, 1, 3 )
+ *
+ * // Major defines a backward compatible series of versions
+ * MALI_MODULE_ASSERT_MAJOR_EQUALS( MODULEZ, 1 )
+ * @endcode
+ *
+ * @par Version Assertions
+ *
+ * This module provides the following compile time version assertion macros.
+ *
+ * @li #MALI_MODULE_ASSERT_MAJOR_EQUALS_MINOR_AT_LEAST
+ * @li #MALI_MODULE_ASSERT_MAJOR_EQUALS
+ * @li #MALI_MODULE_ASSERT_EQUALS
+ * @li #MALI_MODULE_ASSERT_MINIMUM
+ * @li #MALI_MODULE_ASSERT_MAXIMUM
+ *
+ * @par Limitations
+ *
+ * To allow the macros to be placed in the global scope and report more readable
+ * errors, they produce a static function. This makes them unsuitable for use
+ * within headers as the names are only unique on the name of the module under test,
+ * the line number of the current file, and assert type (min, max, equals, ...).
+ */
+
+/**
+ * @addtogroup malisw_version
+ * @{
+ */
+
+#include "arm_cstd/arm_cstd.h"
+
+/**
+ * Private helper macro, indirection so that __LINE__ resolves correctly.
+ */
+#define MALIP_MODULE_ASSERT_FUNCTION_SIGNATURE2( module, type, line ) \
+ static INLINE void _mali_module_##module##_version_check_##type##_##line(void)
+
+#define MALIP_MODULE_ASSERT_FUNCTION_SIGNATURE( module, type, line ) \
+ MALIP_MODULE_ASSERT_FUNCTION_SIGNATURE2( module, type, line )
+
+/**
+ * @hideinitializer
+ * This macro provides a compile time assert that a module interface that has been
+ * @#included in the source base has is greater than or equal to the version specified.
+ *
+ * Expected use is for cases where a module version before the requested minimum
+ * does not support a specific function or is missing an enum affecting the code that is
+ * importing the module.
+ *
+ * It should be invoked at the global scope and ideally following straight after
+ * the module header has been included. For example:
+ *
+ * @code
+ * #include <modulex/mali_modulex.h>
+ *
+ * MALI_MODULE_ASSERT_MINIMUM( MODULEX, 1, 3 )
+ * @endcode
+ */
+#define MALI_MODULE_ASSERT_MINIMUM( module, major, minor ) \
+ MALIP_MODULE_ASSERT_FUNCTION_SIGNATURE( module, minimum, __LINE__ ) \
+ { \
+ CSTD_COMPILE_TIME_ASSERT( ( ( MALI_MODULE_##module##_MAJOR << 16 ) | MALI_MODULE_##module##_MINOR ) \
+ >= ( ( (major) << 16 ) | (minor) ) ); \
+ }
+
+/**
+ * @hideinitializer
+ * This macro provides a compile time assert that a module interface that has been
+ * @#included in the source base is less than or equal to the version specified.
+ *
+ * Expected use is for cases where a later published minor version is found to be
+ * incompatible in some way after the new minor has been issued.
+ *
+ * It should be invoked at the global scope and ideally following straight after
+ * the module header has been included. For example:
+ *
+ * @code
+ * #include <modulex/mali_modulex.h>
+ *
+ * MALI_MODULE_ASSERT_MAXIMUM( MODULEX, 1, 3 )
+ * @endcode
+ */
+#define MALI_MODULE_ASSERT_MAXIMUM( module, major, minor ) \
+ MALIP_MODULE_ASSERT_FUNCTION_SIGNATURE( module, maximum, __LINE__ ) \
+ { \
+ CSTD_COMPILE_TIME_ASSERT( ( ( MALI_MODULE_##module##_MAJOR << 16 ) | MALI_MODULE_##module##_MINOR ) \
+ <= ( ( (major) << 16 ) | (minor) ) ); \
+ }
+
+/**
+ * @hideinitializer
+ * This macro provides a compile time assert that a module interface that has been
+ * @#included in the source base is equal to the version specified.
+ *
+ * Expected use is for cases where a specific version is known to work and other
+ * versions are considered to be risky.
+ *
+ * It should be invoked at the global scope and ideally following straight after
+ * the module header has been included. For example:
+ *
+ * @code
+ * #include <modulex/mali_modulex.h>
+ *
+ * MALI_MODULE_ASSERT_EQUALS( MODULEX, 1, 3 )
+ * @endcode
+ */
+#define MALI_MODULE_ASSERT_EQUALS( module, major, minor ) \
+ MALIP_MODULE_ASSERT_FUNCTION_SIGNATURE( module, equals, __LINE__ ) \
+ { \
+ CSTD_COMPILE_TIME_ASSERT( MALI_MODULE_##module##_MAJOR == major ); \
+ CSTD_COMPILE_TIME_ASSERT( MALI_MODULE_##module##_MINOR == minor ); \
+ }
+
+/**
+ * @hideinitializer
+ * This macro provides a compile time assert that a module interface that has been
+ * @#included in the source base has a major version equal to the major version specified.
+ *
+ * Expected use is for cases where a module is considered low risk and any minor changes
+ * are not considered to be important.
+ *
+ * It should be invoked at the global scope and ideally following straight after
+ * the module header has been included. For example:
+ *
+ * @code
+ * #include <modulex/mali_modulex.h>
+ *
+ * MALI_MODULE_ASSERT_MAJOR_EQUALS( MODULEX, 1, 3 )
+ * @endcode
+ */
+#define MALI_MODULE_ASSERT_MAJOR_EQUALS( module, major ) \
+ MALIP_MODULE_ASSERT_FUNCTION_SIGNATURE( module, major_equals, __LINE__ ) \
+ { \
+ CSTD_COMPILE_TIME_ASSERT( MALI_MODULE_##module##_MAJOR == major ); \
+ }
+
+/**
+ * @hideinitializer
+ * This macro provides a compile time assert that a module interface that has been
+ * @#included in the source base has a major version equal to the major version specified
+ * and that the minor version is at least that which is specified.
+ *
+ * Expected use is for cases where a major revision is suitable from a specific minor
+ * revision but future major versions are a risk.
+ *
+ * It should be invoked at the global scope and ideally following straight after
+ * the module header has been included. For example:
+ *
+ * @code
+ * #include <modulex/mali_modulex.h>
+ *
+ * MALI_MODULE_ASSERT_MAJOR_EQUALS_MINOR_AT_LEAST( MODULEX, 1, 3 )
+ * @endcode
+ */
+#define MALI_MODULE_ASSERT_MAJOR_EQUALS_MINOR_AT_LEAST( module, major, minor ) \
+ MALIP_MODULE_ASSERT_FUNCTION_SIGNATURE( module, major_equals_minor_at_least, __LINE__ ) \
+ { \
+ CSTD_COMPILE_TIME_ASSERT( MALI_MODULE_##module##_MAJOR == major ); \
+ CSTD_COMPILE_TIME_ASSERT( MALI_MODULE_##module##_MINOR >= minor ); \
+ }
+
+/* @} */
+
+/* @} */
+
+#endif /* _MALISW_VERSION_MACROS_H_ */
--- /dev/null
+obj-y += common/
+obj-y += linux/
--- /dev/null
+# Copyright:
+# ----------------------------------------------------------------------------
+# This confidential and proprietary software may be used only as authorized
+# by a licensing agreement from ARM Limited.
+# (C) COPYRIGHT 2010 ARM Limited, ALL RIGHTS RESERVED
+# The entire notice above must be reproduced on all authorized copies and
+# copies may only be made to the extent permitted by a licensing agreement
+# from ARM Limited.
+# ----------------------------------------------------------------------------
+#
+EXTRA_CFLAGS += -I$(ROOT) -I$(ROOT)/osk/src/linux/include -I$(ROOT)/uk/platform_$(PLATFORM)
--- /dev/null
+ccflags-$(CONFIG_VITHAR) += -DMALI_DEBUG=1 -DMALI_HW_TYPE=2 \
+ -DMALI_USE_UMP=0 -DMALI_HW_VERSION=r0p0 \
+ -DMALI_BASE_TRACK_MEMLEAK=0 -DMALI_ANDROID=1 -DMALI_ERROR_INJECT_ON=0 \
+ -DMALI_NO_MALI=0 -DMALI_BACKEND_KERNEL=1 \
+ -DMALI_FAKE_PLATFORM_DEVICE=1 -DMALI_MOCK_TEST=0 -DMALI_KERNEL_TEST_API=0 \
+ -DMALI_INFINITE_CACHE=0 -DMALI_LICENSE_IS_GPL=1 \
+ -DMALI_PLATFORM_CONFIG=exynos5 -DUMP_SVN_REV_STRING="\"dummy\"" \
+ -DMALI_RELEASE_NAME="\"dummy\"" -DMALI_UNIT_TEST=0 -DMALI_INSTRUMENTATION_LEVEL=0 -DMALI_CUSTOMER_RELEASE=1
+
+ROOTDIR = $(src)/../../..
+
+ccflags-$(CONFIG_VITHAR) += -I$(ROOTDIR)/kbase
+ccflags-$(CONFIG_VITHAR) += -I$(src)/..
+
+ccflags-y += -I$(ROOTDIR) -I$(ROOTDIR)/include -I$(ROOTDIR)/osk/src/linux/include -I$(ROOTDIR)/uk/platform_dummy
+ccflags-y += -I$(ROOTDIR)/kbase/midg_gpus/r0p0
+
+obj-y += mali_kbase_device.o
+obj-y += mali_kbase_mem.o
+obj-y += mali_kbase_mmu.o
+obj-y += mali_kbase_jd.o
+obj-y += mali_kbase_jm.o
+obj-y += mali_kbase_gpuprops.o
+obj-y += mali_kbase_js.o
+obj-y += mali_kbase_js_affinity.o
+obj-y += mali_kbase_pm.o
+obj-y += mali_kbase_event.o
+obj-y += mali_kbase_context.o
+obj-y += mali_kbase_pm.o
+obj-y += mali_kbase_pm_driver.o
+obj-y += mali_kbase_pm_metrics.o
+obj-y += mali_kbase_pm_always_on.o
+obj-y += mali_kbase_pm_demand.o
+obj-y += mali_kbase_config.o
+obj-y += mali_kbase_security.o
+obj-y += mali_kbase_instr.o
+#obj-y += mali_kbase_instr_7115.o
+obj-y += mali_kbase_cpuprops.o
+obj-y += mali_kbase_js_ctx_attr.o
+obj-y += mali_kbase_8401_workaround.o
+obj-y += mali_kbase_cache_policy.o
+obj-y += mali_kbase_softjobs.o
+
+obj-y += mali_kbase_pm_metrics_dummy.o
+
+obj-y += mali_kbase_js_policy_cfs.o
+obj-y += mali_kbase_hw.o
+#obj-y += mali_kbase_js_policy_fcfs.o
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _KBASE_H_
+#define _KBASE_H_
+
+#include <malisw/mali_malisw.h>
+#include <osk/mali_osk.h>
+#include <uk/mali_ukk.h>
+
+#include <kbase/mali_base_kernel.h>
+#include <kbase/src/common/mali_kbase_uku.h>
+
+#include "mali_kbase_pm.h"
+#include "mali_kbase_cpuprops.h"
+#include "mali_kbase_gpuprops.h"
+
+#if CSTD_OS_LINUX_KERNEL
+#include <kbase/src/linux/mali_kbase_linux.h>
+#elif defined(MALI_KBASE_USERSPACE)
+#include <kbase/src/userspace/mali_kbase_userspace.h>
+#else
+#error "Unsupported OS"
+#endif
+
+#ifndef KBASE_OS_SUPPORT
+#error Please fix for your platform!
+#endif
+
+#include "mali_kbase_defs.h"
+
+#include "mali_kbase_js.h"
+
+#include "mali_kbase_mem.h"
+
+#include "mali_kbase_security.h"
+
+/**
+ * @page page_base_kernel_main Kernel-side Base (KBase) APIs
+ *
+ * The Kernel-side Base (KBase) APIs are divided up as follows:
+ * - @subpage page_kbase_js_policy
+ */
+
+/**
+ * @defgroup base_kbase_api Kernel-side Base (KBase) APIs
+ */
+
+extern const kbase_device_info kbase_dev_info[];
+
+kbase_device *kbase_device_alloc(void);
+/* note: configuration attributes member of kbdev needs to have been setup before calling kbase_device_init */
+mali_error kbase_device_init(kbase_device *kbdev, const kbase_device_info *dev_info);
+void kbase_device_term(kbase_device *kbdev);
+void kbase_device_free(kbase_device *kbdev);
+int kbase_device_has_feature(kbase_device *kbdev, u32 feature);
+kbase_midgard_type kbase_device_get_type(kbase_device *kbdev);
+
+/**
+ * Ensure that all IRQ handlers have completed execution
+ *
+ * @param kbdev The kbase device
+ */
+void kbase_synchronize_irqs(kbase_device *kbdev);
+
+struct kbase_context *kbase_create_context(kbase_device *kbdev);
+void kbase_destroy_context(kbase_context *kctx);
+mali_error kbase_context_set_create_flags(kbase_context *kctx, u32 flags);
+
+mali_error kbase_instr_hwcnt_setup(kbase_context * kctx, kbase_uk_hwcnt_setup * setup);
+mali_error kbase_instr_hwcnt_enable(kbase_context * kctx, kbase_uk_hwcnt_setup * setup);
+mali_error kbase_instr_hwcnt_disable(kbase_context * kctx);
+mali_error kbase_instr_hwcnt_clear(kbase_context * kctx);
+mali_error kbase_instr_hwcnt_dump(kbase_context * kctx);
+mali_error kbase_instr_hwcnt_dump_irq(kbase_context * kctx);
+mali_bool kbase_instr_hwcnt_dump_complete(kbase_context * kctx, mali_bool *success);
+
+void kbase_clean_caches_done(kbase_device *kbdev);
+
+/**
+ * The GPU has completed performance count sampling successfully.
+ */
+void kbase_instr_hwcnt_sample_done(kbase_device *kbdev);
+
+mali_error kbase_create_os_context(kbase_os_context *osctx);
+void kbase_destroy_os_context(kbase_os_context *osctx);
+
+mali_error kbase_jd_init(struct kbase_context *kctx);
+void kbase_jd_exit(struct kbase_context *kctx);
+mali_error kbase_jd_submit(struct kbase_context *kctx, const kbase_uk_job_submit *user_bag);
+void kbase_jd_post_external_resources(kbase_jd_atom * katom);
+void kbase_jd_done(kbase_jd_atom *katom, int slot_nr, kbasep_js_tick *end_timestamp, mali_bool start_new_jobs);
+void kbase_jd_cancel(kbase_jd_atom *katom);
+void kbase_jd_flush_workqueues(kbase_context *kctx);
+void kbase_jd_zap_context(kbase_context *kctx);
+
+mali_error kbase_job_slot_init(kbase_device *kbdev);
+void kbase_job_slot_halt(kbase_device *kbdev);
+void kbase_job_slot_term(kbase_device *kbdev);
+void kbase_job_done(kbase_device *kbdev, u32 done);
+void kbase_job_zap_context(kbase_context *kctx);
+
+void kbase_job_slot_softstop(kbase_device *kbdev, int js, kbase_jd_atom *target_katom);
+void kbase_job_slot_hardstop(kbase_context *kctx, int js, kbase_jd_atom *target_katom);
+
+void kbase_event_post(kbase_context *ctx, kbase_event *event);
+int kbase_event_dequeue(kbase_context *ctx, base_jd_event *uevent);
+int kbase_event_pending(kbase_context *ctx);
+mali_error kbase_event_init(kbase_context *kctx);
+void kbase_event_close(kbase_context *kctx);
+void kbase_event_cleanup(kbase_context *kctx);
+void kbase_event_wakeup(kbase_context *kctx);
+
+void kbase_process_soft_job( kbase_context *kctx, kbase_jd_atom *katom );
+
+/* api used internally for register access. Contains validation and tracing */
+void kbase_reg_write(kbase_device *kbdev, u16 offset, u32 value, kbase_context * kctx);
+u32 kbase_reg_read(kbase_device *kbdev, u16 offset, kbase_context * kctx);
+void kbase_device_trace_register_access(kbase_context * kctx, kbase_reg_access_type type, u16 reg_offset, u32 reg_value);
+void kbase_device_trace_buffer_install(kbase_context * kctx, u32 * tb, size_t size);
+void kbase_device_trace_buffer_uninstall(kbase_context * kctx);
+
+/* api to be ported per OS, only need to do the raw register access */
+void kbase_os_reg_write(kbase_device *kbdev, u16 offset, u32 value);
+u32 kbase_os_reg_read(kbase_device *kbdev, u16 offset);
+
+/** Report a GPU fault.
+ *
+ * This function is called from the interrupt handler when a GPU fault occurs.
+ * It reports the details of the fault using OSK_PRINT_WARN.
+ *
+ * @param kbdev The kbase device that the GPU fault occurred from.
+ * @param multiple Zero if only GPU_FAULT was raised, non-zero if MULTIPLE_GPU_FAULTS was also set
+ */
+void kbase_report_gpu_fault(kbase_device *kbdev, int multiple);
+
+/** Kill all jobs that are currently running from a context
+ *
+ * This is used in response to a page fault to remove all jobs from the faulting context from the hardware.
+ *
+ * @param kctx The context to kill jobs from
+ */
+void kbase_job_kill_jobs_from_context(kbase_context *kctx);
+
+/**
+ * GPU interrupt handler
+ *
+ * This function is called from the interrupt handler when a GPU irq is to be handled.
+ *
+ * @param kbdev The kbase device to handle an IRQ for
+ * @param val The value of the GPU IRQ status register which triggered the call
+ */
+void kbase_gpu_interrupt(kbase_device * kbdev, u32 val);
+
+/**
+ * Prepare for resetting the GPU.
+ * This function just soft-stops all the slots to ensure that as many jobs as possible are saved.
+ *
+ * The function returns a boolean which should be interpreted as follows:
+ * - MALI_TRUE - Prepared for reset, kbase_reset_gpu should be called.
+ * - MALI_FALSE - Another thread is performing a reset, kbase_reset_gpu should not be called.
+ *
+ * @return See description
+ */
+mali_bool kbase_prepare_to_reset_gpu(kbase_device *kbdev);
+
+/** Reset the GPU
+ *
+ * This function should be called after kbase_prepare_to_reset_gpu iff it returns MALI_TRUE.
+ * It should never be called without a corresponding call to kbase_prepare_to_reset_gpu.
+ *
+ * After this function is called (or not called if kbase_prepare_to_reset_gpu returned MALI_FALSE),
+ * the caller should wait for kbdev->reset_waitq to be signalled to know when the reset has completed.
+ */
+void kbase_reset_gpu(kbase_device *kbdev);
+
+
+/** Returns the name associated with a Mali exception code
+ *
+ * @param exception_code[in] exception code
+ * @return name associated with the exception code
+ */
+const char *kbase_exception_name(u32 exception_code);
+
+
+#if KBASE_TRACE_ENABLE != 0
+/** Add trace values about a job-slot
+ *
+ * @note Any functions called through this macro will still be evaluated in
+ * Release builds (MALI_DEBUG=0). Therefore, when KBASE_TRACE_ENABLE == 0 any
+ * functions called to get the parameters supplied to this macro must:
+ * - be static or static inline
+ * - must just return 0 and have no other statements present in the body.
+ */
+#define KBASE_TRACE_ADD_SLOT( kbdev, code, ctx, uatom, gpu_addr, jobslot ) \
+ kbasep_trace_add( kbdev, KBASE_TRACE_CODE(code), ctx, uatom, gpu_addr, \
+ KBASE_TRACE_FLAG_JOBSLOT, 0, jobslot, 0 )
+
+/** Add trace values about a job-slot, with info
+ *
+ * @note Any functions called through this macro will still be evaluated in
+ * Release builds (MALI_DEBUG=0). Therefore, when KBASE_TRACE_ENABLE == 0 any
+ * functions called to get the parameters supplied to this macro must:
+ * - be static or static inline
+ * - must just return 0 and have no other statements present in the body.
+ */
+#define KBASE_TRACE_ADD_SLOT_INFO( kbdev, code, ctx, uatom, gpu_addr, jobslot, info_val ) \
+ kbasep_trace_add( kbdev, KBASE_TRACE_CODE(code), ctx, uatom, gpu_addr, \
+ KBASE_TRACE_FLAG_JOBSLOT, 0, jobslot, info_val )
+
+
+/** Add trace values about a ctx refcount
+ *
+ * @note Any functions called through this macro will still be evaluated in
+ * Release builds (MALI_DEBUG=0). Therefore, when KBASE_TRACE_ENABLE == 0 any
+ * functions called to get the parameters supplied to this macro must:
+ * - be static or static inline
+ * - must just return 0 and have no other statements present in the body.
+ */
+#define KBASE_TRACE_ADD_REFCOUNT( kbdev, code, ctx, uatom, gpu_addr, refcount ) \
+ kbasep_trace_add( kbdev, KBASE_TRACE_CODE(code), ctx, uatom, gpu_addr, \
+ KBASE_TRACE_FLAG_REFCOUNT, refcount, 0, 0 )
+/** Add trace values about a ctx refcount, and info
+ *
+ * @note Any functions called through this macro will still be evaluated in
+ * Release builds (MALI_DEBUG=0). Therefore, when KBASE_TRACE_ENABLE == 0 any
+ * functions called to get the parameters supplied to this macro must:
+ * - be static or static inline
+ * - must just return 0 and have no other statements present in the body.
+ */
+#define KBASE_TRACE_ADD_REFCOUNT_INFO( kbdev, code, ctx, uatom, gpu_addr, refcount, info_val ) \
+ kbasep_trace_add( kbdev, KBASE_TRACE_CODE(code), ctx, uatom, gpu_addr, \
+ KBASE_TRACE_FLAG_REFCOUNT, refcount, 0, info_val )
+
+/** Add trace values (no slot or refcount)
+ *
+ * @note Any functions called through this macro will still be evaluated in
+ * Release builds (MALI_DEBUG=0). Therefore, when KBASE_TRACE_ENABLE == 0 any
+ * functions called to get the parameters supplied to this macro must:
+ * - be static or static inline
+ * - must just return 0 and have no other statements present in the body.
+ */
+#define KBASE_TRACE_ADD( kbdev, code, ctx, uatom, gpu_addr, info_val ) \
+ kbasep_trace_add( kbdev, KBASE_TRACE_CODE(code), ctx, uatom, gpu_addr, \
+ 0, 0, 0, info_val )
+
+/** Clear the trace */
+#define KBASE_TRACE_CLEAR( kbdev ) \
+ kbasep_trace_clear( kbdev )
+
+/** Dump the slot trace */
+#define KBASE_TRACE_DUMP( kbdev ) \
+ kbasep_trace_dump( kbdev )
+
+/** PRIVATE - do not use directly. Use KBASE_TRACE_ADD() instead */
+void kbasep_trace_add(kbase_device *kbdev, kbase_trace_code code, void *ctx, void *uatom, u64 gpu_addr,
+ u8 flags, int refcount, int jobslot, u32 info_val );
+/** PRIVATE - do not use directly. Use KBASE_TRACE_CLEAR() instead */
+void kbasep_trace_clear(kbase_device *kbdev);
+#else /* KBASE_TRACE_ENABLE != 0 */
+#define KBASE_TRACE_ADD_SLOT( kbdev, code, ctx, uatom, gpu_addr, jobslot )\
+ do{\
+ CSTD_UNUSED(kbdev);\
+ CSTD_NOP(code);\
+ CSTD_UNUSED(ctx);\
+ CSTD_UNUSED(uatom);\
+ CSTD_UNUSED(gpu_addr);\
+ CSTD_UNUSED(jobslot);\
+ }while(0)
+
+#define KBASE_TRACE_ADD_SLOT_INFO( kbdev, code, ctx, uatom, gpu_addr, jobslot, info_val )\
+ do{\
+ CSTD_UNUSED(kbdev);\
+ CSTD_NOP(code);\
+ CSTD_UNUSED(ctx);\
+ CSTD_UNUSED(uatom);\
+ CSTD_UNUSED(gpu_addr);\
+ CSTD_UNUSED(jobslot);\
+ CSTD_UNUSED(info_val);\
+ CSTD_NOP(0);\
+ }while(0)
+
+#define KBASE_TRACE_ADD_REFCOUNT( kbdev, code, ctx, uatom, gpu_addr, refcount )\
+ do{\
+ CSTD_UNUSED(kbdev);\
+ CSTD_NOP(code);\
+ CSTD_UNUSED(ctx);\
+ CSTD_UNUSED(uatom);\
+ CSTD_UNUSED(gpu_addr);\
+ CSTD_UNUSED(refcount);\
+ CSTD_NOP(0);\
+ }while(0)
+
+#define KBASE_TRACE_ADD_REFCOUNT_INFO( kbdev, code, ctx, uatom, gpu_addr, refcount, info_val )\
+ do{\
+ CSTD_UNUSED(kbdev);\
+ CSTD_NOP(code);\
+ CSTD_UNUSED(ctx);\
+ CSTD_UNUSED(uatom);\
+ CSTD_UNUSED(gpu_addr);\
+ CSTD_UNUSED(info_val);\
+ CSTD_NOP(0);\
+ }while(0)
+
+#define KBASE_TRACE_ADD( kbdev, code, subcode, ctx, uatom, val )\
+ do{\
+ CSTD_UNUSED(kbdev);\
+ CSTD_NOP(code);\
+ CSTD_UNUSED(subcode);\
+ CSTD_UNUSED(ctx);\
+ CSTD_UNUSED(uatom);\
+ CSTD_UNUSED(val);\
+ CSTD_NOP(0);\
+ }while(0)
+
+#define KBASE_TRACE_CLEAR( kbdev )\
+ do{\
+ CSTD_UNUSED(kbdev);\
+ CSTD_NOP(0);\
+ }while(0)
+#define KBASE_TRACE_DUMP( kbdev )\
+ do{\
+ CSTD_UNUSED(kbdev);\
+ CSTD_NOP(0);\
+ }while(0)
+
+#endif /* KBASE_TRACE_ENABLE != 0 */
+/** PRIVATE - do not use directly. Use KBASE_TRACE_DUMP() instead */
+void kbasep_trace_dump(kbase_device *kbdev);
+#endif
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_8401_workaround.c
+ * Functions related to working around BASE_HW_ISSUE_8401
+ */
+
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_kbase_defs.h>
+#include <kbase/src/common/mali_kbase_jm.h>
+#include <kbase/src/common/mali_kbase_8401_workaround.h>
+
+#define WORKAROUND_PAGE_OFFSET (2)
+#define URT_POINTER_INDEX (20)
+#define RMU_POINTER_INDEX (23)
+#define RSD_POINTER_INDEX (24)
+#define TSD_POINTER_INDEX (31)
+
+static const u32 compute_job_32bit_header[] =
+{
+ /* Job Descriptor Header */
+
+ /* Job Status */
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+ /* Flags and Indices */
+ /* job_type = compute shader job */
+ 0x00000008, 0x00000000,
+ /* Pointer to next job */
+ 0x00000000,
+ /* Reserved */
+ 0x00000000,
+ /* Job Dimension Data */
+ 0x0000000f, 0x21040842,
+ /* Task Split */
+ 0x08000000,
+ /* Reserved */
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+
+ /* Draw Call Descriptor - 32 bit (Must be aligned to a 64-byte boundry) */
+
+ /* Flags */
+ 0x00000004,
+ /* Primary Attribute Offset */
+ 0x00000000,
+ /* Primitive Index Base Value */
+ 0x00000000,
+
+ /* Pointer To Vertex Position Array (64-byte alignment) */
+ 0x00000000,
+ /* Pointer To Uniform Remapping Table (8-byte alignment) */
+ 0,
+ /* Pointer To Image Descriptor Pointer Table */
+ 0x00000000,
+ /* Pointer To Sampler Array */
+ 0x00000000,
+ /* Pointer To Register-Mapped Uniform Data Area (16-byte alignment) */
+ 0,
+ /* Pointer To Renderer State Descriptor (64-byte alignment) */
+ 0,
+ /* Pointer To Primary Attribute Buffer Array */
+ 0x00000000,
+ /* Pointer To Primary Attribute Array */
+ 0x00000000,
+ /* Pointer To Secondary Attribute Buffer Array */
+ 0x00000000,
+ /* Pointer To Secondary Attribute Array */
+ 0x00000000,
+ /* Pointer To Viewport Descriptor */
+ 0x00000000,
+ /* Pointer To Occlusion Query Result */
+ 0x00000000,
+ /* Pointer To Thread Storage (64 byte alignment) */
+ 0,
+};
+
+
+static const u32 compute_job_32bit_urt[] =
+{
+ /* Uniform Remapping Table Entry */
+ 0, 0,
+};
+
+
+static const u32 compute_job_32bit_rmu[] =
+{
+ /* Register Mapped Uniform Data Area (16 byte aligned), an array of 128-bit
+ * register values.
+ *
+ * NOTE: this is also used as the URT pointer, so the first 16-byte entry
+ * must be all zeros.
+ *
+ * For BASE_HW_ISSUE_8987, we place 16 RMUs here, because this should only
+ * be run concurrently with other GLES jobs (i.e. FS jobs from slot 0).
+ */
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+ 0x00000000, 0x00000000, 0x00000000, 0x00000000
+};
+
+static const u32 compute_job_32bit_rsd[] =
+{
+ /* Renderer State Descriptor */
+
+ /* Shader program inital PC (low) */
+ 0x00000001,
+ /* Shader program initial PC (high) */
+ 0x00000000,
+ /* Image descriptor array sizes */
+ 0x00000000,
+ /* Attribute array sizes */
+ 0x00000000,
+ /* Uniform array size and Shader Flags */
+ /* Flags set: R, D, SE, Reg Uniforms==16, FPM==OpenCL */
+ 0x42003800,
+ /* Depth bias */
+ 0x00000000,
+ /* Depth slope bias */
+ 0x00000000,
+ /* Depth bias clamp */
+ 0x00000000,
+ /* Multisample Write Mask and Flags */
+ 0x00000000,
+ /* Stencil Write Masks and Alpha parameters */
+ 0x00000000,
+ /* Stencil tests - forward facing */
+ 0x00000000,
+ /* Stencel tests - back facing */
+ 0x00000000,
+ /* Alpha Test Reference Value */
+ 0x00000000,
+ /* Thread Balancing Information */
+ 0x00000000,
+ /* Blend Parameters or Pointer (low) */
+ 0x00000000,
+ /* Blend Parameters or Pointer (high) */
+ 0x00000000,
+};
+
+static const u32 compute_job_32bit_tsd[] =
+{
+ /* Thread Storage Descriptor */
+
+ /* Thread Local Storage Sizes */
+ 0x00000000,
+ /* Workgroup Local Memory Area Flags */
+ 0x0000001f,
+ /* Pointer to Local Storage Area */
+ 0x00021000, 0x00000001,
+ /* Pointer to Workgroup Local Storage Area */
+ 0x00000000, 0x00000000,
+ /* Pointer to Shader Exception Handler */
+ 0x00000000, 0x00000000
+};
+
+static kbase_jd_atom dummy_job_atom[KBASE_8401_WORKAROUND_COMPUTEJOB_COUNT];
+
+/**
+ * Initialize the compute job sturcture.
+ */
+
+static void kbasep_8401_workaround_update_job_pointers(u32 *dummy_compute_job, int page_nr)
+{
+ u32 base_address = (page_nr+WORKAROUND_PAGE_OFFSET)*OSK_PAGE_SIZE;
+ u8 *dummy_job = (u8*) dummy_compute_job;
+ u8 *dummy_job_urt;
+ u8 *dummy_job_rmu;
+ u8 *dummy_job_rsd;
+ u8 *dummy_job_tsd;
+
+ OSK_ASSERT(dummy_compute_job);
+
+ /* determin where each job section goes taking alignment restrictions into consideration */
+ dummy_job_urt = (u8*) ((((uintptr_t)dummy_job + sizeof(compute_job_32bit_header))+7) & ~7);
+ dummy_job_rmu = (u8*) ((((uintptr_t)dummy_job_urt + sizeof(compute_job_32bit_urt))+15) & ~15);
+ dummy_job_rsd = (u8*) ((((uintptr_t)dummy_job_rmu + sizeof(compute_job_32bit_rmu))+63) & ~63);
+ dummy_job_tsd = (u8*) ((((uintptr_t)dummy_job_rsd + sizeof(compute_job_32bit_rsd))+63) & ~63);
+
+ /* Make sure the job fits within a single page */
+ OSK_ASSERT(OSK_PAGE_SIZE > ((dummy_job_tsd+sizeof(compute_job_32bit_tsd)) - dummy_job));
+
+ /* Copy the job sections to the allocated memory */
+ memcpy(dummy_job, compute_job_32bit_header, sizeof(compute_job_32bit_header));
+ memcpy(dummy_job_urt, compute_job_32bit_urt, sizeof(compute_job_32bit_urt));
+ memcpy(dummy_job_rmu, compute_job_32bit_rmu, sizeof(compute_job_32bit_rmu));
+ memcpy(dummy_job_rsd, compute_job_32bit_rsd, sizeof(compute_job_32bit_rsd));
+ memcpy(dummy_job_tsd, compute_job_32bit_tsd, sizeof(compute_job_32bit_tsd));
+
+ /* Update header pointers */
+ *(dummy_compute_job + URT_POINTER_INDEX) = (dummy_job_urt - dummy_job) + base_address;
+ *(dummy_compute_job + RMU_POINTER_INDEX) = (dummy_job_rmu - dummy_job) + base_address;
+ *(dummy_compute_job + RSD_POINTER_INDEX) = (dummy_job_rsd - dummy_job) + base_address;
+ *(dummy_compute_job + TSD_POINTER_INDEX) = (dummy_job_tsd - dummy_job) + base_address;
+ /* Update URT pointer */
+ *((u32*)dummy_job_urt+0) = (((dummy_job_rmu - dummy_job) + base_address) << 8) & 0xffffff00;
+ *((u32*)dummy_job_urt+1) = (((dummy_job_rmu - dummy_job) + base_address) >> 24) & 0xff;
+}
+
+/**
+ * Initialize the memory for 8401 workaround.
+ */
+
+mali_error kbasep_8401_workaround_init(kbase_device *kbdev)
+{
+ kbasep_js_device_data *js_devdata;
+ kbase_context *workaround_kctx;
+ u32 count;
+ int i;
+ u16 as_present_mask;
+
+ OSK_ASSERT(kbdev);
+ OSK_ASSERT(kbdev->workaround_kctx == NULL);
+
+ js_devdata = &kbdev->js_data;
+
+ /* For this workaround we reserve one address space to allow us to
+ * submit a special job independent of other contexts */
+ --(kbdev->nr_hw_address_spaces);
+
+ if ( kbdev->nr_user_address_spaces == (kbdev->nr_hw_address_spaces + 1) )
+ {
+ /* Only update nr_user_address_spaces if it was unchanged - to ensure
+ * HW workarounds that have modified this will still work */
+ --(kbdev->nr_user_address_spaces);
+ }
+ OSK_ASSERT( kbdev->nr_user_address_spaces <= kbdev->nr_hw_address_spaces );
+
+ /* Recalculate the free address spaces bit-pattern */
+ as_present_mask = (1U << kbdev->nr_hw_address_spaces) - 1;
+ js_devdata->as_free &= as_present_mask;
+
+ workaround_kctx = kbase_create_context(kbdev);
+ if(!workaround_kctx)
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ /* Allocate the pages required to contain the job */
+ count = kbase_phy_pages_alloc(workaround_kctx->kbdev,
+ &workaround_kctx->pgd_allocator,
+ KBASE_8401_WORKAROUND_COMPUTEJOB_COUNT,
+ kbdev->workaround_compute_job_pa);
+ if(count < KBASE_8401_WORKAROUND_COMPUTEJOB_COUNT)
+ {
+ goto page_release;
+ }
+
+ /* Get virtual address of mapped memory and write a compute job for each page */
+ for(i = 0; i < KBASE_8401_WORKAROUND_COMPUTEJOB_COUNT; i++)
+ {
+ kbdev->workaround_compute_job_va[i] = osk_kmap(kbdev->workaround_compute_job_pa[i]);
+ if(NULL == kbdev->workaround_compute_job_va[i])
+ {
+ goto page_free;
+ }
+
+ /* Generate the compute job data */
+ kbasep_8401_workaround_update_job_pointers((u32*)kbdev->workaround_compute_job_va[i], i);
+ }
+
+ /* Insert pages to the gpu mmu. */
+ kbase_mmu_insert_pages(workaround_kctx,
+ /* vpfn = page number */
+ (u64)WORKAROUND_PAGE_OFFSET,
+ /* physical address */
+ kbdev->workaround_compute_job_pa,
+ /* number of pages */
+ KBASE_8401_WORKAROUND_COMPUTEJOB_COUNT,
+ /* flags */
+ KBASE_REG_GPU_RD|KBASE_REG_CPU_RD|KBASE_REG_CPU_WR|KBASE_REG_GPU_WR);
+
+ kbdev->workaround_kctx = workaround_kctx;
+ return MALI_ERROR_NONE;
+page_free:
+ while(i--)
+ {
+ osk_kunmap(kbdev->workaround_compute_job_pa[i], kbdev->workaround_compute_job_va[i]);
+ }
+page_release:
+ kbase_phy_pages_free(kbdev, &workaround_kctx->pgd_allocator, count, kbdev->workaround_compute_job_pa);
+ kbase_destroy_context(workaround_kctx);
+
+ return MALI_ERROR_FUNCTION_FAILED;
+}
+
+/**
+ * Free up the memory used by 8401 workaround.
+ **/
+
+void kbasep_8401_workaround_term(kbase_device *kbdev)
+{
+ kbasep_js_device_data *js_devdata;
+ int i;
+ u16 restored_as;
+
+ OSK_ASSERT(kbdev);
+ OSK_ASSERT(kbdev->workaround_kctx);
+
+ js_devdata = &kbdev->js_data;
+
+ for(i = 0; i < KBASE_8401_WORKAROUND_COMPUTEJOB_COUNT; i++)
+ {
+ osk_kunmap(kbdev->workaround_compute_job_pa[i], kbdev->workaround_compute_job_va[i]);
+ }
+
+ kbase_phy_pages_free(kbdev, &kbdev->workaround_kctx->pgd_allocator, KBASE_8401_WORKAROUND_COMPUTEJOB_COUNT, kbdev->workaround_compute_job_pa);
+
+ kbase_destroy_context(kbdev->workaround_kctx);
+ kbdev->workaround_kctx = NULL;
+
+ /* Free up the workaround address space */
+ kbdev->nr_hw_address_spaces++;
+
+ if ( kbdev->nr_user_address_spaces == (kbdev->nr_hw_address_spaces - 1) )
+ {
+ /* Only update nr_user_address_spaces if it was unchanged - to ensure
+ * HW workarounds that have modified this will still work */
+ ++(kbdev->nr_user_address_spaces);
+ }
+ OSK_ASSERT( kbdev->nr_user_address_spaces <= kbdev->nr_hw_address_spaces );
+
+ /* Recalculate the free address spaces bit-pattern */
+ restored_as = (1U << kbdev->nr_hw_address_spaces);
+ js_devdata->as_free |= restored_as;
+}
+
+/**
+ * Submit the 8401 workaround job.
+ *
+ * Important for BASE_HW_ISSUE_8987: This job always uses 16 RMUs
+ * - Therefore, on slot[1] it will always use the same number of RMUs as another
+ * GLES job.
+ * - On slot[2], no other job (GLES or otherwise) will be running on the
+ * cores, by virtue of it being slot[2]. Therefore, any value of RMUs is
+ * acceptable.
+ */
+void kbasep_8401_submit_dummy_job(kbase_device *kbdev, int js)
+{
+ u32 cfg;
+ mali_addr64 jc;
+ u32 pgd_high;
+
+ /* While this workaround is active we reserve the last address space just for submitting the dummy jobs */
+ int as = kbdev->nr_hw_address_spaces;
+
+ /* Don't issue compute jobs on job slot 0 */
+ OSK_ASSERT(js != 0);
+ OSK_ASSERT(js < KBASE_8401_WORKAROUND_COMPUTEJOB_COUNT);
+
+ /* Job chain GPU address */
+ jc = (js+WORKAROUND_PAGE_OFFSET)*OSK_PAGE_SIZE; /* GPU phys address (see kbase_mmu_insert_pages call in kbasep_8401_workaround_init*/
+
+ /* Clear the job status words which may contain values from a previous job completion */
+ memset(kbdev->workaround_compute_job_va[js], 0, 4*sizeof(u32));
+
+ /* Get the affinity of the previous job */
+ dummy_job_atom[js].affinity = ((u64)kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_AFFINITY_LO), NULL)) |
+ (((u64)kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_AFFINITY_HI), NULL)) << 32);
+
+ /* Don't submit a compute job if the affinity was previously zero (i.e. no jobs have run yet on this slot) */
+ if(!dummy_job_atom[js].affinity)
+ {
+ return;
+ }
+
+ /* Ensure that our page tables are programmed into the MMU */
+ kbase_reg_write(kbdev, MMU_AS_REG(as, ASn_TRANSTAB_LO),
+ (kbdev->workaround_kctx->pgd & ASn_TRANSTAB_ADDR_SPACE_MASK) | ASn_TRANSTAB_READ_INNER
+ | ASn_TRANSTAB_ADRMODE_TABLE, NULL);
+
+ /* Need to use a conditional expression to avoid "right shift count >= width of type"
+ * error when using an if statement - although the size_of condition is evaluated at compile
+ * time the unused branch is not removed until after it is type-checked and the error
+ * produced.
+ */
+ pgd_high = sizeof(kbdev->workaround_kctx->pgd) > 4 ? (kbdev->workaround_kctx->pgd >> 32) : 0;
+ kbase_reg_write(kbdev, MMU_AS_REG(as, ASn_TRANSTAB_HI), pgd_high, NULL);
+
+ kbase_reg_write(kbdev, MMU_AS_REG(as, ASn_MEMATTR_LO), ASn_MEMATTR_IMPL_DEF_CACHE_POLICY, NULL);
+ kbase_reg_write(kbdev, MMU_AS_REG(as, ASn_MEMATTR_HI), ASn_MEMATTR_IMPL_DEF_CACHE_POLICY, NULL);
+ kbase_reg_write(kbdev, MMU_AS_REG(as, ASn_COMMAND), ASn_COMMAND_UPDATE, NULL);
+
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_LO), jc & 0xFFFFFFFF, NULL);
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_HI), jc >> 32, NULL);
+
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_AFFINITY_NEXT_LO), dummy_job_atom[js].affinity & 0xFFFFFFFF, NULL);
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_AFFINITY_NEXT_HI), dummy_job_atom[js].affinity >> 32, NULL);
+
+ /* start MMU, medium priority, cache clean/flush on end, clean/flush on start */
+ cfg = as | JSn_CONFIG_END_FLUSH_CLEAN_INVALIDATE | JSn_CONFIG_START_MMU
+ | JSn_CONFIG_START_FLUSH_CLEAN_INVALIDATE | JSn_CONFIG_THREAD_PRI(8);
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_CONFIG_NEXT), cfg, NULL);
+
+ KBASE_TRACE_ADD_SLOT( kbdev, JM_SUBMIT, NULL, 0, jc, js );
+
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_COMMAND_NEXT), JSn_COMMAND_START, NULL);
+ /* Report that the job has been submitted */
+ kbasep_jm_enqueue_submit_slot(&kbdev->jm_slots[js], &dummy_job_atom[js]);
+}
+
+/**
+ * Check if the katom given is a dummy compute job.
+ */
+mali_bool kbasep_8401_is_workaround_job(kbase_jd_atom *katom)
+{
+ int i;
+
+ /* Note: we don't check the first dummy_job_atom as slot 0 is never used for the workaround */
+ for(i = 1; i < KBASE_8401_WORKAROUND_COMPUTEJOB_COUNT; i++)
+ {
+ if(katom == &dummy_job_atom[i])
+ {
+ /* This is a dummy job */
+ return MALI_TRUE;
+ }
+ }
+
+ /* This is a real job */
+ return MALI_FALSE;
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_8401_workaround.h
+ * Functions related to working around BASE_HW_ISSUE_8401
+ */
+
+#ifndef _KBASE_8401_WORKAROUND_H_
+#define _KBASE_8401_WORKAROUND_H_
+
+mali_error kbasep_8401_workaround_init(kbase_device *kbdev);
+void kbasep_8401_workaround_term(kbase_device *kbdev);
+void kbasep_8401_submit_dummy_job(kbase_device *kbdev, int js);
+mali_bool kbasep_8401_is_workaround_job(kbase_jd_atom *katom);
+
+#endif /* _KBASE_8401_WORKAROUND_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_cache_policy.h
+ * Cache Policy API.
+ */
+
+#include "mali_kbase_cache_policy.h"
+
+/*
+ * The output flags should be a combination of the following values:
+ * KBASE_REG_CPU_CACHED: CPU cache should be enabled
+ * KBASE_REG_GPU_CACHED: GPU cache should be enabled
+ *
+ * The input flags may contain a combination of hints:
+ * BASE_MEM_HINT_CPU_RD: region heavily read CPU side
+ * BASE_MEM_HINT_CPU_WR: region heavily written CPU side
+ * BASE_MEM_HINT_GPU_RD: region heavily read GPU side
+ * BASE_MEM_HINT_GPU_WR: region heavily written GPU side
+ */
+u32 kbase_cache_enabled(u32 flags, u32 nr_pages)
+{
+ u32 cache_flags = 0;
+
+ CSTD_UNUSED(nr_pages);
+
+ /* The CPU cache should be enabled for regions heavily read and written
+ * from the CPU side
+ */
+#if !MALI_UNCACHED
+ if ((flags & BASE_MEM_HINT_CPU_RD) && (flags & BASE_MEM_HINT_CPU_WR))
+ {
+ cache_flags |= KBASE_REG_CPU_CACHED;
+ }
+#endif
+
+ /* The GPU cache should be enabled for regions heavily read and written
+ * from the GPU side
+ */
+ if ((flags & BASE_MEM_HINT_GPU_RD) && (flags & BASE_MEM_HINT_GPU_WR))
+ {
+ cache_flags |= KBASE_REG_GPU_CACHED;
+ }
+
+ return cache_flags;
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_cache_policy.h
+ * Cache Policy API.
+ */
+
+#ifndef _KBASE_CACHE_POLICY_H_
+#define _KBASE_CACHE_POLICY_H_
+
+#include <malisw/mali_malisw.h>
+#include "mali_kbase.h"
+#include <kbase/mali_base_kernel.h>
+
+/**
+ * @brief Choose the cache policy for a specific region
+ *
+ * Tells whether the CPU and GPU caches should be enabled or not for a specific region.
+ * This function can be modified to customize the cache policy depending on the flags
+ * and size of the region.
+ *
+ * @param[in] flags flags describing attributes of the region
+ * @param[in] size total number of pages (backed or not) for the region
+ *
+ * @return a combination of KBASE_REG_CPU_CACHED and KBASE_REG_GPU_CACHED depending
+ * on the cache policy
+ */
+u32 kbase_cache_enabled(u32 flags, u32 nr_pages);
+
+#endif /* _KBASE_CACHE_POLICY_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_kbase_defs.h>
+#include <kbase/src/common/mali_kbase_cpuprops.h>
+#include <osk/mali_osk.h>
+#if MALI_USE_UMP == 1
+#include <ump/ump_common.h>
+#endif /* MALI_USE_UMP == 1 */
+
+/* Specifies how many attributes are permitted in the config (excluding terminating attribute).
+ * This is used in validation function so we can detect if configuration is properly terminated. This value can be
+ * changed if we need to introduce more attributes or many memory regions need to be defined */
+#define ATTRIBUTE_COUNT_MAX 32
+
+/* right now we allow only 2 memory attributes (excluding termination attribute) */
+#define MEMORY_ATTRIBUTE_COUNT_MAX 2
+
+/* Limits for gpu frequency configuration parameters. These will use for config validation. */
+#define MAX_GPU_ALLOWED_FREQ_KHZ 1000000
+#define MIN_GPU_ALLOWED_FREQ_KHZ 1
+
+/* Default irq throttle time. This is the default desired minimum time in between two consecutive
+ * interrupts from the gpu. The irq throttle gpu register is set after this value. */
+#define DEFAULT_IRQ_THROTTLE_TIME_US 20
+
+/*** Begin Scheduling defaults ***/
+
+/**
+ * Default scheduling tick granuality, in nanoseconds
+ */
+#define DEFAULT_JS_SCHEDULING_TICK_NS 100000000u /* 100ms */
+
+/**
+ * Default minimum number of scheduling ticks before jobs are soft-stopped.
+ *
+ * This defines the time-slice for a job (which may be different from that of a context)
+ */
+#define DEFAULT_JS_SOFT_STOP_TICKS 1 /* Between 0.1 and 0.2s before soft-stop */
+
+/**
+ * Default minimum number of scheduling ticks before Soft-Stoppable
+ * (BASE_JD_REQ_NSS bit clear) jobs are hard-stopped
+ */
+#define DEFAULT_JS_HARD_STOP_TICKS_SS_HW_ISSUE_8408 12 /* 1.2s before hard-stop, for a certain GLES2 test at 128x128 (bound by combined vertex+tiler job) */
+#define DEFAULT_JS_HARD_STOP_TICKS_SS 2 /* Between 0.2 and 0.3s before hard-stop */
+
+/**
+ * Default minimum number of scheduling ticks before Non-Soft-Stoppable
+ * (BASE_JD_REQ_NSS bit set) jobs are hard-stopped
+ */
+#define DEFAULT_JS_HARD_STOP_TICKS_NSS 600 /* 60s @ 100ms tick */
+
+/**
+ * Default minimum number of scheduling ticks before the GPU is reset
+ * to clear a "stuck" Soft-Stoppable job
+ */
+#define DEFAULT_JS_RESET_TICKS_SS_HW_ISSUE_8408 18 /* 1.8s before resetting GPU, for a certain GLES2 test at 128x128 (bound by combined vertex+tiler job) */
+#define DEFAULT_JS_RESET_TICKS_SS 3 /* 0.3-0.4s before GPU is reset */
+
+/**
+ * Default minimum number of scheduling ticks before the GPU is reset
+ * to clear a "stuck" Non-Soft-Stoppable job
+ */
+#define DEFAULT_JS_RESET_TICKS_NSS 601 /* 60.1s @ 100ms tick */
+
+/**
+ * Number of milliseconds given for other jobs on the GPU to be
+ * soft-stopped when the GPU needs to be reset.
+ */
+#define DEFAULT_JS_RESET_TIMEOUT_MS 3000
+
+/**
+ * Default timeslice that a context is scheduled in for, in nanoseconds.
+ *
+ * When a context has used up this amount of time across its jobs, it is
+ * scheduled out to let another run.
+ *
+ * @note the resolution is nanoseconds (ns) here, because that's the format
+ * often used by the OS.
+ */
+#define DEFAULT_JS_CTX_TIMESLICE_NS 50000000 /* 0.05s - at 20fps a ctx does at least 1 frame before being scheduled out. At 40fps, 2 frames, etc */
+
+/**
+ * Default initial runtime of a context for CFS, in ticks.
+ *
+ * This value is relative to that of the least-run context, and defines where
+ * in the CFS queue a new context is added.
+ */
+#define DEFAULT_JS_CFS_CTX_RUNTIME_INIT_SLICES 1
+
+/**
+ * Default minimum runtime value of a context for CFS, in ticks.
+ *
+ * This value is relative to that of the least-run context. This prevents
+ * "stored-up timeslices" DoS attacks.
+ */
+#define DEFAULT_JS_CFS_CTX_RUNTIME_MIN_SLICES 2
+
+/**
+ * Default setting for whether to prefer security or performance.
+ *
+ * Currently affects only r0p0-15dev0 HW and earlier.
+ */
+#define DEFAULT_SECURE_BUT_LOSS_OF_PERFORMANCE MALI_FALSE
+
+/*** End Scheduling defaults ***/
+
+/**
+ * Default value for KBASE_CONFIG_ATTR_CPU_SPEED_FUNC.
+ * Points to @ref kbase_cpuprops_get_default_clock_speed.
+ */
+#define DEFAULT_CPU_SPEED_FUNC ((uintptr_t)kbase_cpuprops_get_default_clock_speed)
+
+#if (!defined(MALI_KBASE_USERSPACE) || !MALI_KBASE_USERSPACE) && (!MALI_LICENSE_IS_GPL || MALI_FAKE_PLATFORM_DEVICE)
+
+extern kbase_platform_config platform_config;
+kbase_platform_config *kbasep_get_platform_config(void)
+{
+ return &platform_config;
+}
+#endif /* (!defined(MALI_KBASE_USERSPACE) || !MALI_KBASE_USERSPACE) && (!MALI_LICENSE_IS_GPL || MALI_FAKE_PLATFORM_DEVICE) */
+
+#
+
+int kbasep_get_config_attribute_count(const kbase_attribute *attributes)
+{
+ int count = 1;
+
+ OSK_ASSERT(attributes != NULL);
+
+ while (attributes->id != KBASE_CONFIG_ATTR_END)
+ {
+ attributes++;
+ count++;
+ }
+
+ return count;
+}
+
+int kbasep_get_config_attribute_count_by_id(const kbase_attribute *attributes, int attribute_id)
+{
+ int count = 0;
+ OSK_ASSERT(attributes != NULL);
+
+ while (attributes->id != KBASE_CONFIG_ATTR_END)
+ {
+ if (attributes->id == attribute_id)
+ {
+ count++;
+ }
+ attributes++;
+ }
+
+ return count;
+}
+
+static const char* midgard_type_strings[] =
+ {
+ "mali-t6xm",
+ "mali-t6f1",
+ "mali-t601",
+ "mali-t604",
+ "mali-t608"
+ };
+
+const char *kbasep_midgard_type_to_string(kbase_midgard_type midgard_type)
+{
+ OSK_ASSERT(midgard_type < KBASE_MALI_COUNT);
+
+ return midgard_type_strings[midgard_type];
+}
+
+const kbase_attribute *kbasep_get_next_attribute(const kbase_attribute *attributes, int attribute_id)
+{
+ OSK_ASSERT(attributes != NULL);
+
+ while (attributes->id != KBASE_CONFIG_ATTR_END)
+ {
+ if (attributes->id == attribute_id)
+ {
+ return attributes;
+ }
+ attributes++;
+ }
+ return NULL;
+}
+KBASE_EXPORT_TEST_API(kbasep_get_next_attribute)
+
+uintptr_t kbasep_get_config_value(struct kbase_device *kbdev, const kbase_attribute *attributes, int attribute_id)
+{
+ const kbase_attribute *attr;
+
+ OSK_ASSERT(attributes != NULL);
+
+ attr = kbasep_get_next_attribute(attributes, attribute_id);
+ if (attr != NULL)
+ {
+ return attr->data;
+ }
+
+ /* default values */
+ switch (attribute_id)
+ {
+ case KBASE_CONFIG_ATTR_MEMORY_PER_PROCESS_LIMIT:
+ return (uintptr_t)-1;
+#if MALI_USE_UMP == 1
+ case KBASE_CONFIG_ATTR_UMP_DEVICE:
+ return UMP_DEVICE_W_SHIFT;
+#endif /* MALI_USE_UMP == 1 */
+ case KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_MAX:
+ return (uintptr_t)-1;
+ case KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_PERF_GPU:
+ return KBASE_MEM_PERF_NORMAL;
+ case KBASE_CONFIG_ATTR_GPU_IRQ_THROTTLE_TIME_US:
+ return DEFAULT_IRQ_THROTTLE_TIME_US;
+ /* Begin scheduling defaults */
+ case KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS:
+ return DEFAULT_JS_SCHEDULING_TICK_NS;
+ case KBASE_CONFIG_ATTR_JS_SOFT_STOP_TICKS:
+ return DEFAULT_JS_SOFT_STOP_TICKS;
+ case KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS:
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8408))
+ {
+ return DEFAULT_JS_HARD_STOP_TICKS_SS_HW_ISSUE_8408;
+ }
+ else
+ {
+ return DEFAULT_JS_HARD_STOP_TICKS_SS;
+ }
+ case KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS:
+ return DEFAULT_JS_HARD_STOP_TICKS_NSS;
+ case KBASE_CONFIG_ATTR_JS_CTX_TIMESLICE_NS:
+ return DEFAULT_JS_CTX_TIMESLICE_NS;
+ case KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_INIT_SLICES:
+ return DEFAULT_JS_CFS_CTX_RUNTIME_INIT_SLICES;
+ case KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_MIN_SLICES:
+ return DEFAULT_JS_CFS_CTX_RUNTIME_MIN_SLICES;
+ case KBASE_CONFIG_ATTR_JS_RESET_TICKS_SS:
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8408))
+ {
+ return DEFAULT_JS_RESET_TICKS_SS_HW_ISSUE_8408;
+ }
+ else
+ {
+ return DEFAULT_JS_RESET_TICKS_SS;
+ }
+ case KBASE_CONFIG_ATTR_JS_RESET_TICKS_NSS:
+ return DEFAULT_JS_RESET_TICKS_NSS;
+ case KBASE_CONFIG_ATTR_JS_RESET_TIMEOUT_MS:
+ return DEFAULT_JS_RESET_TIMEOUT_MS;
+ /* End scheduling defaults */
+ case KBASE_CONFIG_ATTR_POWER_MANAGEMENT_CALLBACKS:
+ return 0;
+ case KBASE_CONFIG_ATTR_PLATFORM_FUNCS:
+ return 0;
+ case KBASE_CONFIG_ATTR_SECURE_BUT_LOSS_OF_PERFORMANCE:
+ return DEFAULT_SECURE_BUT_LOSS_OF_PERFORMANCE;
+ case KBASE_CONFIG_ATTR_CPU_SPEED_FUNC:
+ return DEFAULT_CPU_SPEED_FUNC;
+ default:
+ OSK_PRINT_ERROR(OSK_BASE_CORE,
+ "kbasep_get_config_value. Cannot get value of attribute with id=%d and no default value defined",
+ attribute_id);
+ return 0;
+ }
+}
+KBASE_EXPORT_TEST_API(kbasep_get_config_value)
+
+mali_bool kbasep_platform_device_init(kbase_device *kbdev)
+{
+ kbase_platform_funcs_conf *platform_funcs;
+
+ platform_funcs = (kbase_platform_funcs_conf *) kbasep_get_config_value(kbdev, kbdev->config_attributes, KBASE_CONFIG_ATTR_PLATFORM_FUNCS);
+ if(platform_funcs)
+ {
+ if(platform_funcs->platform_init_func)
+ {
+ return platform_funcs->platform_init_func(kbdev);
+ }
+ }
+ return MALI_TRUE;
+}
+
+void kbasep_platform_device_term(kbase_device *kbdev)
+{
+ kbase_platform_funcs_conf *platform_funcs;
+
+ platform_funcs = (kbase_platform_funcs_conf *) kbasep_get_config_value(kbdev, kbdev->config_attributes, KBASE_CONFIG_ATTR_PLATFORM_FUNCS);
+ if(platform_funcs)
+ {
+ if(platform_funcs->platform_term_func)
+ {
+ platform_funcs->platform_term_func(kbdev);
+ }
+ }
+}
+
+void kbasep_get_memory_performance(const kbase_memory_resource *resource, kbase_memory_performance *cpu_performance,
+ kbase_memory_performance *gpu_performance)
+{
+ kbase_attribute *attributes;
+
+ OSK_ASSERT(resource != NULL);
+ OSK_ASSERT(cpu_performance != NULL );
+ OSK_ASSERT(gpu_performance != NULL);
+
+ attributes = resource->attributes;
+ *cpu_performance = *gpu_performance = KBASE_MEM_PERF_NORMAL; /* default performance */
+
+ if (attributes == NULL)
+ {
+ return;
+ }
+
+ while (attributes->id != KBASE_CONFIG_ATTR_END)
+ {
+ if (attributes->id == KBASE_MEM_ATTR_PERF_GPU)
+ {
+ *gpu_performance = (kbase_memory_performance) attributes->data;
+ }
+ else if (attributes->id == KBASE_MEM_ATTR_PERF_CPU)
+ {
+ *cpu_performance = (kbase_memory_performance) attributes->data;
+ }
+ attributes++;
+ }
+}
+
+#if MALI_USE_UMP == 1
+static mali_bool kbasep_validate_ump_device(int ump_device)
+{
+ mali_bool valid;
+
+ switch (ump_device)
+ {
+ case UMP_DEVICE_W_SHIFT:
+ case UMP_DEVICE_X_SHIFT:
+ case UMP_DEVICE_Y_SHIFT:
+ case UMP_DEVICE_Z_SHIFT:
+ valid = MALI_TRUE;
+ break;
+ default:
+ valid = MALI_FALSE;
+ break;
+ }
+ return valid;
+}
+#endif /* MALI_USE_UMP == 1 */
+
+static mali_bool kbasep_validate_memory_performance(kbase_memory_performance performance)
+{
+ return performance <= KBASE_MEM_PERF_MAX_VALUE;
+}
+
+static mali_bool kbasep_validate_memory_resource(const kbase_memory_resource *memory_resource)
+{
+ OSK_ASSERT(memory_resource != NULL);
+
+ if (memory_resource->name == NULL)
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Unnamed memory region found");
+ return MALI_FALSE;
+ }
+
+ if (memory_resource->base & ((1 << OSK_PAGE_SHIFT) - 1))
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Base address of \"%s\" memory region is not page aligned", memory_resource->name);
+ return MALI_FALSE;
+ }
+
+ if (memory_resource->size & ((1 << OSK_PAGE_SHIFT) - 1))
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Size of \"%s\" memory region is not a multiple of page size", memory_resource->name);
+ return MALI_FALSE;
+ }
+
+ if (memory_resource->attributes != NULL) /* we allow NULL attribute list */
+ {
+ int i;
+
+ for (i = 0; memory_resource->attributes[i].id != KBASE_MEM_ATTR_END; i++)
+ {
+ if (i >= MEMORY_ATTRIBUTE_COUNT_MAX)
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "More than MEMORY_ATTRIBUTE_COUNT_MAX=%d configuration attributes defined. Is memory attribute list properly terminated?",
+ MEMORY_ATTRIBUTE_COUNT_MAX);
+ return MALI_FALSE;
+ }
+ switch(memory_resource->attributes[i].id)
+ {
+ case KBASE_MEM_ATTR_PERF_CPU:
+ if (MALI_TRUE != kbasep_validate_memory_performance(
+ (kbase_memory_performance)memory_resource->attributes[i].data))
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "CPU performance of \"%s\" region is invalid: %d",
+ memory_resource->name, (kbase_memory_performance)memory_resource->attributes[i].data);
+ return MALI_FALSE;
+ }
+ break;
+
+ case KBASE_MEM_ATTR_PERF_GPU:
+ if (MALI_TRUE != kbasep_validate_memory_performance(
+ (kbase_memory_performance)memory_resource->attributes[i].data))
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "GPU performance of \"%s\" region is invalid: %d",
+ memory_resource->name, (kbase_memory_performance)memory_resource->attributes[i].data);
+ return MALI_FALSE;
+ }
+ break;
+ default:
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Invalid memory attribute found in \"%s\" memory region: %d",
+ memory_resource->name, memory_resource->attributes[i].id);
+ return MALI_FALSE;
+ }
+ }
+ }
+
+ return MALI_TRUE;
+}
+
+
+static mali_bool kbasep_validate_gpu_clock_freq(kbase_device *kbdev, const kbase_attribute *attributes)
+{
+ uintptr_t freq_min = kbasep_get_config_value(kbdev, attributes, KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MIN);
+ uintptr_t freq_max = kbasep_get_config_value(kbdev, attributes, KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MAX);
+
+ if ((freq_min > MAX_GPU_ALLOWED_FREQ_KHZ) ||
+ (freq_min < MIN_GPU_ALLOWED_FREQ_KHZ) ||
+ (freq_max > MAX_GPU_ALLOWED_FREQ_KHZ) ||
+ (freq_max < MIN_GPU_ALLOWED_FREQ_KHZ) ||
+ (freq_min > freq_max))
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Invalid GPU frequencies found in configuration: min=%ldkHz, max=%ldkHz.", freq_min, freq_max);
+ return MALI_FALSE;
+ }
+
+ return MALI_TRUE;
+}
+
+static mali_bool kbasep_validate_pm_callback(const kbase_pm_callback_conf *callbacks)
+{
+ if (callbacks == NULL)
+ {
+ /* Having no callbacks is valid */
+ return MALI_TRUE;
+ }
+ if ((callbacks->power_off_callback != NULL && callbacks->power_on_callback == NULL) ||
+ (callbacks->power_off_callback == NULL && callbacks->power_on_callback != NULL))
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Invalid power management callbacks: Only one of power_off_callback and power_on_callback was specified");
+ return MALI_FALSE;
+ }
+ return MALI_TRUE;
+}
+
+static mali_bool kbasep_validate_cpu_speed_func(kbase_cpuprops_clock_speed_function fcn)
+{
+ return fcn != NULL;
+}
+
+mali_bool kbasep_validate_configuration_attributes(kbase_device *kbdev, const kbase_attribute *attributes)
+{
+ int i;
+ mali_bool had_gpu_freq_min = MALI_FALSE, had_gpu_freq_max = MALI_FALSE;
+
+ OSK_ASSERT(attributes);
+
+ for (i = 0; attributes[i].id != KBASE_CONFIG_ATTR_END; i++)
+ {
+ if (i >= ATTRIBUTE_COUNT_MAX)
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "More than ATTRIBUTE_COUNT_MAX=%d configuration attributes defined. Is attribute list properly terminated?",
+ ATTRIBUTE_COUNT_MAX);
+ return MALI_FALSE;
+ }
+
+ switch (attributes[i].id)
+ {
+ case KBASE_CONFIG_ATTR_MEMORY_RESOURCE:
+ if (MALI_FALSE == kbasep_validate_memory_resource((kbase_memory_resource *)attributes[i].data))
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Invalid memory region found in configuration");
+ return MALI_FALSE;
+ }
+ break;
+
+ case KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_MAX:
+ /* Some shared memory is required for GPU page tables, see MIDBASE-1534 */
+ if ( 0 == attributes[i].data )
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Maximum OS Shared Memory Maximum is set to 0 which is not supported");
+ return MALI_FALSE;
+ }
+ break;
+
+ case KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_PERF_GPU:
+ if (MALI_FALSE == kbasep_validate_memory_performance((kbase_memory_performance)attributes[i].data))
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Shared OS memory GPU performance attribute has invalid value: %d",
+ (kbase_memory_performance)attributes[i].data);
+ return MALI_FALSE;
+ }
+ break;
+
+ case KBASE_CONFIG_ATTR_MEMORY_PER_PROCESS_LIMIT:
+ /* any value is allowed */
+ break;
+#if MALI_USE_UMP == 1
+ case KBASE_CONFIG_ATTR_UMP_DEVICE:
+ if (MALI_FALSE == kbasep_validate_ump_device(attributes[i].data))
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Unknown UMP device found in configuration: %d",
+ (int)attributes[i].data);
+ return MALI_FALSE;
+ }
+ break;
+#endif /* MALI_USE_UMP == 1 */
+
+ case KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MIN:
+ had_gpu_freq_min = MALI_TRUE;
+ if (MALI_FALSE == kbasep_validate_gpu_clock_freq(kbdev, attributes))
+ {
+ /* Warning message handled by kbasep_validate_gpu_clock_freq() */
+ return MALI_FALSE;
+ }
+ break;
+
+ case KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MAX:
+ had_gpu_freq_max = MALI_TRUE;
+ if (MALI_FALSE == kbasep_validate_gpu_clock_freq(kbdev, attributes))
+ {
+ /* Warning message handled by kbasep_validate_gpu_clock_freq() */
+ return MALI_FALSE;
+ }
+ break;
+
+ /* Only non-zero unsigned 32-bit values accepted */
+ case KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS:
+ #if CSTD_CPU_64BIT
+ if ( attributes[i].data == 0u || (u64)attributes[i].data > (u64)U32_MAX )
+ #else
+ if ( attributes[i].data == 0u )
+ #endif
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Invalid Job Scheduling Configuration attribute for "
+ "KBASE_CONFIG_ATTR_JS_SCHEDULING_TICKS_NS: %d",
+ (int)attributes[i].data);
+ return MALI_FALSE;
+ }
+ break;
+
+ /* All these Job Scheduling attributes are FALLTHROUGH: only unsigned 32-bit values accepted */
+ case KBASE_CONFIG_ATTR_JS_SOFT_STOP_TICKS:
+ case KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS:
+ case KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS:
+ case KBASE_CONFIG_ATTR_JS_RESET_TICKS_SS:
+ case KBASE_CONFIG_ATTR_JS_RESET_TICKS_NSS:
+ case KBASE_CONFIG_ATTR_JS_RESET_TIMEOUT_MS:
+ case KBASE_CONFIG_ATTR_JS_CTX_TIMESLICE_NS:
+ case KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_INIT_SLICES:
+ case KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_MIN_SLICES:
+ #if CSTD_CPU_64BIT
+ if ( (u64)attributes[i].data > (u64)U32_MAX )
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Job Scheduling Configuration attribute exceeds 32-bits: "
+ "id==%d val==%d",
+ attributes[i].id, (int)attributes[i].data);
+ return MALI_FALSE;
+ }
+ #endif
+ break;
+
+ case KBASE_CONFIG_ATTR_GPU_IRQ_THROTTLE_TIME_US:
+ #if CSTD_CPU_64BIT
+ if ( (u64)attributes[i].data > (u64)U32_MAX )
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "IRQ throttle time attribute exceeds 32-bits: "
+ "id==%d val==%d",
+ attributes[i].id, (int)attributes[i].data);
+ return MALI_FALSE;
+ }
+ #endif
+ break;
+
+ case KBASE_CONFIG_ATTR_POWER_MANAGEMENT_CALLBACKS:
+ if (MALI_FALSE == kbasep_validate_pm_callback((kbase_pm_callback_conf*)attributes[i].data))
+ {
+ /* Warning message handled by kbasep_validate_pm_callback() */
+ return MALI_FALSE;
+ }
+ break;
+
+ case KBASE_CONFIG_ATTR_SECURE_BUT_LOSS_OF_PERFORMANCE:
+ if ( attributes[i].data != MALI_TRUE && attributes[i].data != MALI_FALSE )
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE,
+ "Value for KBASE_CONFIG_ATTR_SECURE_BUT_LOSS_OF_PERFORMANCE was not "
+ "MALI_TRUE or MALI_FALSE: %u",
+ (unsigned int)attributes[i].data);
+ return MALI_FALSE;
+ }
+ break;
+
+ case KBASE_CONFIG_ATTR_CPU_SPEED_FUNC:
+ if (MALI_FALSE == kbasep_validate_cpu_speed_func((kbase_cpuprops_clock_speed_function)attributes[i].data))
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Invalid function pointer in KBASE_CONFIG_ATTR_CPU_SPEED_FUNC");
+ return MALI_FALSE;
+ }
+ break;
+
+ case KBASE_CONFIG_ATTR_PLATFORM_FUNCS:
+ /* any value is allowed */
+ break;
+
+ default:
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Invalid attribute found in configuration: %d", attributes[i].id);
+ return MALI_FALSE;
+ }
+ }
+
+ if(!had_gpu_freq_min)
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Configuration does not include mandatory attribute KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MIN");
+ return MALI_FALSE;
+ }
+
+ if(!had_gpu_freq_max)
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Configuration does not include mandatory attribute KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MAX");
+ return MALI_FALSE;
+ }
+
+ return MALI_TRUE;
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_context.c
+ * Base kernel context APIs
+ */
+
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_midg_regmap.h>
+
+/**
+ * @brief Create a kernel base context.
+ *
+ * Allocate and init a kernel base context. Calls
+ * kbase_create_os_context() to setup OS specific structures.
+ */
+struct kbase_context *kbase_create_context(kbase_device *kbdev)
+{
+ struct kbase_context *kctx;
+ struct kbase_va_region *pmem_reg;
+ struct kbase_va_region *tmem_reg;
+ struct kbase_va_region *exec_reg;
+ osk_error osk_err;
+ mali_error mali_err;
+
+ OSK_ASSERT(kbdev != NULL);
+
+ /* zero-inited as lot of code assume it's zero'ed out on create */
+ kctx = osk_calloc(sizeof(*kctx));
+ if (!kctx)
+ goto out;
+
+ kctx->kbdev = kbdev;
+ kctx->as_nr = KBASEP_AS_NR_INVALID;
+
+ if (kbase_mem_usage_init(&kctx->usage, kctx->kbdev->memdev.per_process_memory_limit >> OSK_PAGE_SHIFT))
+ {
+ goto free_kctx;
+ }
+
+ if (kbase_jd_init(kctx))
+ goto free_memctx;
+
+ mali_err = kbasep_js_kctx_init( kctx );
+ if ( MALI_ERROR_NONE != mali_err )
+ {
+ goto free_jd; /* safe to call kbasep_js_kctx_term in this case */
+ }
+
+ mali_err = kbase_event_init(kctx);
+ if (MALI_ERROR_NONE != mali_err)
+ goto free_jd;
+
+ osk_err = osk_mutex_init(&kctx->reg_lock, OSK_LOCK_ORDER_MEM_REG);
+ if (OSK_ERR_NONE != osk_err)
+ goto free_event;
+
+ OSK_DLIST_INIT(&kctx->reg_list);
+
+ /* Use a new *Shared Memory* allocator for GPU page tables.
+ * See MIDBASE-1534 for details. */
+ osk_err = osk_phy_allocator_init(&kctx->pgd_allocator, 0, 0, NULL);
+ if (OSK_ERR_NONE != osk_err)
+ goto free_region_lock;
+
+ mali_err = kbase_mmu_init(kctx);
+ if(MALI_ERROR_NONE != mali_err)
+ goto free_phy;
+
+ kctx->pgd = kbase_mmu_alloc_pgd(kctx);
+ if (!kctx->pgd)
+ goto free_mmu;
+
+ if (kbase_create_os_context(&kctx->osctx))
+ goto free_pgd;
+
+ kctx->nr_outstanding_atoms = 0;
+ if ( OSK_ERR_NONE != osk_waitq_init(&kctx->complete_outstanding_waitq))
+ {
+ goto free_osctx;
+ }
+ osk_waitq_set(&kctx->complete_outstanding_waitq);
+
+ /* Make sure page 0 is not used... */
+ pmem_reg = kbase_alloc_free_region(kctx, 1,
+ KBASE_REG_ZONE_EXEC_BASE - 1, KBASE_REG_ZONE_PMEM);
+ exec_reg = kbase_alloc_free_region(kctx, KBASE_REG_ZONE_EXEC_BASE,
+ KBASE_REG_ZONE_EXEC_SIZE, KBASE_REG_ZONE_EXEC);
+ tmem_reg = kbase_alloc_free_region(kctx, KBASE_REG_ZONE_TMEM_BASE,
+ KBASE_REG_ZONE_TMEM_SIZE, KBASE_REG_ZONE_TMEM);
+
+ if (!pmem_reg || !exec_reg || !tmem_reg)
+ {
+ if (pmem_reg)
+ kbase_free_alloced_region(pmem_reg);
+ if (exec_reg)
+ kbase_free_alloced_region(exec_reg);
+ if (tmem_reg)
+ kbase_free_alloced_region(tmem_reg);
+
+ kbase_destroy_context(kctx);
+ return NULL;
+ }
+
+ OSK_DLIST_PUSH_FRONT(&kctx->reg_list, pmem_reg, struct kbase_va_region, link);
+ OSK_DLIST_PUSH_BACK(&kctx->reg_list, exec_reg, struct kbase_va_region, link);
+ OSK_DLIST_PUSH_BACK(&kctx->reg_list, tmem_reg, struct kbase_va_region, link);
+
+ return kctx;
+free_osctx:
+ kbase_destroy_os_context(&kctx->osctx);
+free_pgd:
+ kbase_mmu_free_pgd(kctx);
+free_mmu:
+ kbase_mmu_term(kctx);
+free_phy:
+ osk_phy_allocator_term(&kctx->pgd_allocator);
+free_region_lock:
+ osk_mutex_term(&kctx->reg_lock);
+free_event:
+ kbase_event_cleanup(kctx);
+free_jd:
+ /* Safe to call this one even when didn't initialize (assuming kctx was sufficiently zeroed) */
+ kbasep_js_kctx_term(kctx);
+ kbase_jd_exit(kctx);
+free_memctx:
+ kbase_mem_usage_term(&kctx->usage);
+free_kctx:
+ osk_free(kctx);
+out:
+ return NULL;
+
+}
+KBASE_EXPORT_SYMBOL(kbase_create_context)
+
+/**
+ * @brief Destroy a kernel base context.
+ *
+ * Destroy a kernel base context. Calls kbase_destroy_os_context() to
+ * free OS specific structures. Will release all outstanding regions.
+ */
+void kbase_destroy_context(struct kbase_context *kctx)
+{
+ kbase_device *kbdev;
+
+ OSK_ASSERT(NULL != kctx);
+
+ kbdev = kctx->kbdev;
+ OSK_ASSERT(NULL != kbdev);
+
+ KBASE_TRACE_ADD( kbdev, CORE_CTX_DESTROY, kctx, NULL, 0u, 0u );
+
+ /* Ensure the core is powered up for the destroy process */
+ kbase_pm_context_active(kbdev);
+
+ if(kbdev->hwcnt.kctx == kctx)
+ {
+ /* disable the use of the hw counters if the app didn't use the API correctly or crashed */
+ KBASE_TRACE_ADD( kbdev, CORE_CTX_HWINSTR_TERM, kctx, NULL, 0u, 0u );
+ OSK_PRINT_WARN(OSK_BASE_CTX,
+ "The privileged process asking for instrumentation forgot to disable it "
+ "before exiting. Will end instrumentation for them" );
+ kbase_instr_hwcnt_disable(kctx);
+ }
+
+ kbase_jd_zap_context(kctx);
+ kbase_event_cleanup(kctx);
+
+ kbase_gpu_vm_lock(kctx);
+
+ /* MMU is disabled as part of scheduling out the context */
+ kbase_mmu_free_pgd(kctx);
+ osk_phy_allocator_term(&kctx->pgd_allocator);
+ OSK_DLIST_EMPTY_LIST(&kctx->reg_list, struct kbase_va_region,
+ link, kbase_free_alloced_region);
+ kbase_destroy_os_context(&kctx->osctx);
+ kbase_gpu_vm_unlock(kctx);
+
+ /* Safe to call this one even when didn't initialize (assuming kctx was sufficiently zeroed) */
+ kbasep_js_kctx_term(kctx);
+
+ kbase_jd_exit(kctx);
+ osk_mutex_term(&kctx->reg_lock);
+
+ kbase_pm_context_idle(kbdev);
+
+ kbase_mmu_term(kctx);
+
+ kbase_mem_usage_term(&kctx->usage);
+
+ osk_waitq_term(&kctx->complete_outstanding_waitq);
+ osk_free(kctx);
+}
+KBASE_EXPORT_SYMBOL(kbase_destroy_context)
+
+/**
+ * Set creation flags on a context
+ */
+mali_error kbase_context_set_create_flags(kbase_context *kctx, u32 flags)
+{
+ mali_error err = MALI_ERROR_NONE;
+ kbasep_js_kctx_info *js_kctx_info;
+ OSK_ASSERT(NULL != kctx);
+
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ /* Validate flags */
+ if ( flags != (flags & BASE_CONTEXT_CREATE_KERNEL_FLAGS) )
+ {
+ err = MALI_ERROR_FUNCTION_FAILED;
+ goto out;
+ }
+
+ osk_mutex_lock( &js_kctx_info->ctx.jsctx_mutex );
+
+ /* Ensure this is the first call */
+ if ( (js_kctx_info->ctx.flags & KBASE_CTX_FLAG_CREATE_FLAGS_SET) != 0 )
+ {
+ OSK_PRINT_ERROR(OSK_BASE_CTX, "User attempted to set context creation flags more than once - not allowed");
+ err = MALI_ERROR_FUNCTION_FAILED;
+ goto out_unlock;
+ }
+
+ js_kctx_info->ctx.flags |= KBASE_CTX_FLAG_CREATE_FLAGS_SET;
+
+ /* Translate the flags */
+ if ( (flags & BASE_CONTEXT_SYSTEM_MONITOR_SUBMIT_DISABLED) == 0 )
+ {
+ /* This flag remains set until it is explicitly cleared */
+ js_kctx_info->ctx.flags &= ~((u32)KBASE_CTX_FLAG_SUBMIT_DISABLED);
+ }
+
+ if ( (flags & BASE_CONTEXT_HINT_ONLY_COMPUTE) != 0 )
+ {
+ js_kctx_info->ctx.flags |= (u32)KBASE_CTX_FLAG_HINT_ONLY_COMPUTE;
+ }
+
+ /* Latch the initial attributes into the Job Scheduler */
+ kbasep_js_ctx_attr_set_initial_attrs( kctx->kbdev, kctx );
+
+
+out_unlock:
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+out:
+ return err;
+}
+KBASE_EXPORT_SYMBOL(kbase_context_set_create_flags)
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_cpuprops.c
+ * Base kernel property query APIs
+ */
+
+#include "mali_kbase_cpuprops.h"
+#include "mali_kbase.h"
+#include "mali_kbase_uku.h"
+#include <kbase/mali_kbase_config.h>
+#include <osk/mali_osk.h>
+
+int kbase_cpuprops_get_default_clock_speed(u32 *clock_speed)
+{
+ OSK_ASSERT( NULL != clock_speed );
+
+ *clock_speed = 100;
+ return 0;
+}
+
+mali_error kbase_cpuprops_uk_get_props(struct kbase_context *kctx, kbase_uk_cpuprops * kbase_props)
+{
+ int result;
+ kbase_cpuprops_clock_speed_function kbase_cpuprops_uk_get_clock_speed;
+
+ kbase_props->props.cpu_l1_dcache_line_size_log2 = OSK_L1_DCACHE_LINE_SIZE_LOG2;
+ kbase_props->props.cpu_l1_dcache_size = OSK_L1_DCACHE_SIZE;
+ kbase_props->props.cpu_flags = BASE_CPU_PROPERTY_FLAG_LITTLE_ENDIAN;
+
+ kbase_props->props.nr_cores = OSK_NUM_CPUS;
+ kbase_props->props.cpu_page_size_log2 = OSK_PAGE_SHIFT;
+ kbase_props->props.available_memory_size = OSK_MEM_PAGES << OSK_PAGE_SHIFT;
+
+ kbase_cpuprops_uk_get_clock_speed = (kbase_cpuprops_clock_speed_function)kbasep_get_config_value( kctx->kbdev, kctx->kbdev->config_attributes, KBASE_CONFIG_ATTR_CPU_SPEED_FUNC );
+ result = kbase_cpuprops_uk_get_clock_speed(&kbase_props->props.max_cpu_clock_speed_mhz);
+ if (result != 0)
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ return MALI_ERROR_NONE;
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_cpuprops.h
+ * Base kernel property query APIs
+ */
+
+#ifndef _KBASE_CPUPROPS_H_
+#define _KBASE_CPUPROPS_H_
+
+#include <malisw/mali_malisw.h>
+
+/* Forward declarations */
+struct kbase_context;
+struct kbase_uk_cpuprops;
+
+/**
+ * @brief Default implementation of @ref KBASE_CONFIG_ATTR_CPU_SPEED_FUNC.
+ *
+ * This function sets clock_speed to 100, so will be an underestimate for
+ * any real system.
+ *
+ * See @refkbase_cpuprops_clock_speed_function for details on the parameters
+ * and return value.
+ */
+int kbase_cpuprops_get_default_clock_speed(u32 *clock_speed);
+
+/**
+ * @brief Provides CPU properties data.
+ *
+ * Fill the kbase_uk_cpuprops with values from CPU configuration.
+ *
+ * @param kctx The kbase context
+ * @param kbase_props A copy of the kbase_uk_cpuprops structure from userspace
+ *
+ * @return MALI_ERROR_NONE on success. Any other value indicates failure.
+ */
+mali_error kbase_cpuprops_uk_get_props(struct kbase_context *kctx, struct kbase_uk_cpuprops* kbase_props);
+
+#endif /*_KBASE_CPUPROPS_H_*/
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_defs.h
+ *
+ * Defintions (types, defines, etcs) common to Kbase. They are placed here to
+ * allow the hierarchy of header files to work.
+ */
+
+#ifndef _KBASE_DEFS_H_
+#define _KBASE_DEFS_H_
+
+#define KBASE_DRV_NAME "mali"
+
+#include <kbase/mali_kbase_config.h>
+#include <kbase/mali_base_hwconfig.h>
+#include <osk/mali_osk.h>
+
+#ifdef CONFIG_KDS
+#include <kds/include/linux/kds.h>
+#endif
+
+/** Enable SW tracing when set */
+#ifndef KBASE_TRACE_ENABLE
+#if MALI_DEBUG
+#define KBASE_TRACE_ENABLE 1
+#else
+#define KBASE_TRACE_ENABLE 0
+#endif /*MALI_DEBUG*/
+#endif /*KBASE_TRACE_ENABLE*/
+
+/* Maximum number of outstanding atoms per kbase context
+ * this is set for security reasons to prevent a malicious app
+ * from hanging the driver */
+#define MAX_KCTX_OUTSTANDING_ATOMS (1ul << 6)
+
+/** Dump Job slot trace on error (only active if KBASE_TRACE_ENABLE != 0) */
+#define KBASE_TRACE_DUMP_ON_JOB_SLOT_ERROR 1
+
+/**
+ * Number of milliseconds before resetting the GPU when a job cannot be "zapped" from the hardware.
+ * Note that the time is actually ZAP_TIMEOUT+SOFT_STOP_RESET_TIMEOUT between the context zap starting and the GPU
+ * actually being reset to give other contexts time for their jobs to be soft-stopped and removed from the hardware
+ * before resetting.
+ */
+#define ZAP_TIMEOUT 1000
+
+/**
+ * Prevent soft-stops from occuring in scheduling situations
+ *
+ * This is not due to HW issues, but when scheduling is desired to be more predictable.
+ *
+ * Therefore, soft stop may still be disabled due to HW issues.
+ *
+ * @note Soft stop will still be used for non-scheduling purposes e.g. when terminating a context.
+ *
+ * @note if not in use, define this value to 0 instead of #undef'ing it
+ */
+#define KBASE_DISABLE_SCHEDULING_SOFT_STOPS 0
+
+/**
+ * Prevent hard-stops from occuring in scheduling situations
+ *
+ * This is not due to HW issues, but when scheduling is desired to be more predictable.
+ *
+ * @note Hard stop will still be used for non-scheduling purposes e.g. when terminating a context.
+ *
+ * @note if not in use, define this value to 0 instead of #undef'ing it
+ */
+#define KBASE_DISABLE_SCHEDULING_HARD_STOPS 0
+
+/* Forward declarations+defintions */
+typedef struct kbase_context kbase_context;
+typedef struct kbase_jd_atom kbasep_jd_atom;
+typedef struct kbase_device kbase_device;
+
+/**
+ * The maximum number of Job Slots to support in the Hardware.
+ *
+ * You can optimize this down if your target devices will only ever support a
+ * small number of job slots.
+ */
+#define BASE_JM_MAX_NR_SLOTS 16
+
+/**
+ * The maximum number of Address Spaces to support in the Hardware.
+ *
+ * You can optimize this down if your target devices will only ever support a
+ * small number of Address Spaces
+ */
+#define BASE_MAX_NR_AS 16
+
+#ifndef UINTPTR_MAX
+
+/**
+ * @brief Maximum value representable by type uintptr_t
+ */
+#if CSTD_CPU_32BIT
+#define UINTPTR_MAX U32_MAX
+#elif CSTD_CPU_64BIT
+#define UINTPTR_MAX U64_MAX
+#endif /* CSTD_CPU_64BIT */
+
+#endif /* !defined(UINTPTR_MAX) */
+
+/* mmu */
+#define ENTRY_IS_ATE 1ULL
+#define ENTRY_IS_INVAL 2ULL
+#define ENTRY_IS_PTE 3ULL
+
+#define MIDGARD_MMU_VA_BITS 48
+
+#define ENTRY_ATTR_BITS (7ULL << 2) /* bits 4:2 */
+#define ENTRY_RD_BIT (1ULL << 6)
+#define ENTRY_WR_BIT (1ULL << 7)
+#define ENTRY_SHARE_BITS (3ULL << 8) /* bits 9:8 */
+#define ENTRY_ACCESS_BIT (1ULL << 10)
+#define ENTRY_NX_BIT (1ULL << 54)
+
+#define ENTRY_FLAGS_MASK (ENTRY_ATTR_BITS | ENTRY_RD_BIT | ENTRY_WR_BIT | ENTRY_SHARE_BITS | ENTRY_ACCESS_BIT | ENTRY_NX_BIT)
+
+#if MIDGARD_MMU_VA_BITS > 39
+#define MIDGARD_MMU_TOPLEVEL 0
+#else
+#define MIDGARD_MMU_TOPLEVEL 1
+#endif
+
+#define GROWABLE_FLAGS_REQUIRED (KBASE_REG_PF_GROW | KBASE_REG_ZONE_TMEM)
+#define GROWABLE_FLAGS_MASK (GROWABLE_FLAGS_REQUIRED | KBASE_REG_FREE)
+
+/** setting in kbase_context::as_nr that indicates it's invalid */
+#define KBASEP_AS_NR_INVALID (-1)
+
+#define KBASE_LOCK_REGION_MAX_SIZE (63)
+#define KBASE_LOCK_REGION_MIN_SIZE (11)
+
+#define KBASE_TRACE_SIZE_LOG2 8 /* 256 entries */
+#define KBASE_TRACE_SIZE (1 << KBASE_TRACE_SIZE_LOG2)
+#define KBASE_TRACE_MASK ((1 << KBASE_TRACE_SIZE_LOG2)-1)
+
+#include "mali_kbase_js_defs.h"
+
+typedef struct kbase_event {
+ osk_dlist_item entry;
+ const void *data;
+ base_jd_event_code event_code;
+} kbase_event;
+
+
+/* Hijack the event entry field to link the struct with the different
+ * queues... */
+typedef struct kbase_jd_bag {
+ kbase_event event;
+ u64 core_restriction;
+ size_t offset;
+ u32 nr_atoms;
+ /** Set when the bag has a power management reference. This is used to ensure that the GPU is
+ * not turned off after a soft-job has read the GPU counters until the bag has completed */
+ mali_bool8 has_pm_ctx_reference;
+} kbase_jd_bag;
+
+/**
+ * @brief States to model state machine processed by kbasep_js_job_check_ref_cores(), which
+ * handles retaining cores for power management and affinity management.
+ *
+ * The state @ref KBASE_ATOM_COREREF_STATE_RECHECK_AFFINITY prevents an attack
+ * where lots of atoms could be submitted before powerup, and each has an
+ * affinity chosen that causes other atoms to have an affinity
+ * violation. Whilst the affinity was not causing violations at the time it
+ * was chosen, it could cause violations thereafter. For example, 1000 jobs
+ * could have had their affinity chosen during the powerup time, so any of
+ * those 1000 jobs could cause an affinity violation later on.
+ *
+ * The attack would otherwise occur because other atoms/contexts have to wait for:
+ * -# the currently running atoms (which are causing the violation) to
+ * finish
+ * -# and, the atoms that had their affinity chosen during powerup to
+ * finish. These are run preferrentially because they don't cause a
+ * violation, but instead continue to cause the violation in others.
+ * -# or, the attacker is scheduled out (which might not happen for just 2
+ * contexts)
+ *
+ * By re-choosing the affinity (which is designed to avoid violations at the
+ * time it's chosen), we break condition (2) of the wait, which minimizes the
+ * problem to just waiting for current jobs to finish (which can be bounded if
+ * the Job Scheduling Policy has a timer).
+ */
+typedef enum
+{
+ /** Starting state: No affinity chosen, and cores must be requested. kbase_jd_atom::affinity==0 */
+ KBASE_ATOM_COREREF_STATE_NO_CORES_REQUESTED,
+ /** Cores requested, but waiting for them to be powered. Requested cores given by kbase_jd_atom::affinity */
+ KBASE_ATOM_COREREF_STATE_WAITING_FOR_REQUESTED_CORES,
+ /** Cores given by kbase_jd_atom::affinity are powered, but affinity might be out-of-date, so must recheck */
+ KBASE_ATOM_COREREF_STATE_RECHECK_AFFINITY,
+ /** Cores given by kbase_jd_atom::affinity are powered, and affinity is up-to-date, but must check for violations */
+ KBASE_ATOM_COREREF_STATE_CHECK_AFFINITY_VIOLATIONS,
+ /** Cores are powered, kbase_jd_atom::affinity up-to-date, no affinity violations: atom can be submitted to HW */
+ KBASE_ATOM_COREREF_STATE_READY
+
+} kbase_atom_coreref_state;
+
+typedef struct kbase_jd_atom {
+ kbase_event event;
+ osk_workq_work work;
+ kbasep_js_tick start_timestamp;
+ base_jd_atom *user_atom;
+ kbase_jd_bag *bag;
+ kbase_context *kctx;
+ base_jd_dep pre_dep;
+ base_jd_dep post_dep;
+ u32 nr_syncsets;
+ u32 nr_extres;
+ u32 device_nr;
+ u64 affinity;
+ u64 jc;
+ kbase_atom_coreref_state coreref_state;
+#ifdef CONFIG_KDS
+ struct kds_resource_set * kds_rset;
+ mali_bool kds_dep_satisfied;
+#endif
+
+ base_jd_core_req core_req; /**< core requirements */
+
+ kbasep_js_policy_job_info sched_info;
+ /** Job Slot to retry submitting to if submission from IRQ handler failed
+ *
+ * NOTE: see if this can be unified into the another member e.g. the event */
+ int retry_submit_on_slot;
+ /* atom priority scaled to nice range with +20 offset 0..39 */
+ int nice_prio;
+
+ int poking; /* BASE_HW_ISSUE_8316 */
+} kbase_jd_atom;
+
+/*
+ * Theory of operations:
+ *
+ * - sem is an array of 256 bits, each bit being a semaphore
+ * for a 1-1 job dependency:
+ * Initially set to 0 (passing)
+ * Incremented when a post_dep is queued
+ * Decremented when a post_dep is completed
+ * pre_dep is satisfied when value is 0
+ * sem #0 is hardwired to 0 (always passing).
+ *
+ * - queue is an array of atoms, one per semaphore.
+ * When a pre_dep is not satisfied, the atom is added to both
+ * queues it depends on (except for queue 0 which is never used).
+ * Each time a post_dep is signal, the corresponding bit is cleared,
+ * the atoms removed from the queue, and the corresponding pre_dep
+ * is cleared. The atom can be run when pre_dep[0] == pre_dep[1] == 0.
+ */
+
+#define KBASE_JD_DEP_QUEUE_SIZE 256
+
+typedef struct kbase_jd_dep_queue {
+ kbase_jd_atom *queue[KBASE_JD_DEP_QUEUE_SIZE];
+ u32 sem[BASEP_JD_SEM_ARRAY_SIZE];
+} kbase_jd_dep_queue;
+
+typedef struct kbase_jd_context {
+ osk_mutex lock;
+ kbasep_js_kctx_info sched_info;
+ kbase_jd_dep_queue dep_queue;
+ base_jd_atom *pool;
+ size_t pool_size;
+
+ /** Tracks all job-dispatch jobs. This includes those not tracked by
+ * the scheduler: 'not ready to run' and 'dependency-only' jobs. */
+ u32 job_nr;
+
+ /** Waitq that reflects whether there are no jobs (including SW-only
+ * dependency jobs). This is set when no jobs are present on the ctx,
+ * and clear when there are jobs.
+ *
+ * @note: Job Dispatcher knows about more jobs than the Job Scheduler:
+ * the Job Scheduler is unaware of jobs that are blocked on dependencies,
+ * and SW-only dependency jobs.
+ *
+ * This waitq can be waited upon to find out when the context jobs are all
+ * done/cancelled (including those that might've been blocked on
+ * dependencies) - and so, whether it can be terminated. However, it should
+ * only be terminated once it is neither present in the policy-queue (see
+ * kbasep_js_policy_try_evict_ctx() ) nor the run-pool (see
+ * kbasep_js_kctx_info::ctx::is_scheduled_waitq).
+ *
+ * Since the waitq is only set under kbase_jd_context::lock,
+ * the waiter should also briefly obtain and drop kbase_jd_context::lock to
+ * guarentee that the setter has completed its work on the kbase_context */
+ osk_waitq zero_jobs_waitq;
+ osk_workq job_done_wq;
+ osk_spinlock_irq tb_lock;
+ u32 *tb;
+ size_t tb_wrap_offset;
+
+#ifdef CONFIG_KDS
+ struct kds_callback kds_cb;
+#endif
+} kbase_jd_context;
+
+typedef struct kbase_jm_slot
+{
+ osk_spinlock_irq lock;
+
+ /* The number of slots must be a power of two */
+#define BASE_JM_SUBMIT_SLOTS 16
+#define BASE_JM_SUBMIT_SLOTS_MASK (BASE_JM_SUBMIT_SLOTS - 1)
+
+ kbase_jd_atom *submitted[BASE_JM_SUBMIT_SLOTS];
+
+ u8 submitted_head;
+ u8 submitted_nr;
+
+} kbase_jm_slot;
+
+typedef enum kbase_midgard_type
+{
+ KBASE_MALI_T6XM,
+ KBASE_MALI_T6F1,
+ KBASE_MALI_T601,
+ KBASE_MALI_T604,
+ KBASE_MALI_T608,
+
+ KBASE_MALI_COUNT
+} kbase_midgard_type;
+
+#define KBASE_FEATURE_HAS_MODEL_PMU (1U << 0)
+#define KBASE_FEATURE_NEEDS_REG_DELAY (1U << 1)
+#define KBASE_FEATURE_HAS_16BIT_PC (1U << 2)
+#define KBASE_FEATURE_LACKS_RESET_INT (1U << 3)
+#define KBASE_FEATURE_DELAYED_PERF_WRITE_STATUS (1U << 4)
+
+typedef struct kbase_device_info
+{
+ kbase_midgard_type dev_type;
+ u32 features;
+} kbase_device_info;
+
+/**
+ * Important: Our code makes assumptions that a kbase_as structure is always at
+ * kbase_device->as[number]. This is used to recover the containing
+ * kbase_device from a kbase_as structure.
+ *
+ * Therefore, kbase_as structures must not be allocated anywhere else.
+ */
+typedef struct kbase_as
+{
+ int number;
+
+ osk_workq pf_wq;
+ osk_workq_work work_pagefault;
+ osk_workq_work work_busfault;
+ mali_addr64 fault_addr;
+ osk_mutex transaction_mutex;
+
+ /* BASE_HW_ISSUE_8316 */
+ osk_workq poke_wq;
+ osk_workq_work poke_work;
+ osk_atomic poke_refcount;
+ osk_timer poke_timer;
+} kbase_as;
+
+/* tracking of memory usage */
+typedef struct kbasep_mem_usage
+{
+ u32 max_pages;
+ osk_atomic cur_pages;
+} kbasep_mem_usage;
+
+/**
+ * @brief Specifies order in which physical allocators are selected.
+ *
+ * Enumeration lists different orders in which physical allocators are selected on memory allocation.
+ *
+ */
+typedef enum kbase_phys_allocator_order
+{
+ ALLOCATOR_ORDER_CONFIG, /* Select allocators in order they appeared in the configuration file */
+ ALLOCATOR_ORDER_GPU_PERFORMANCE, /* Select allocators in order from fastest to slowest on the GPU */
+ ALLOCATOR_ORDER_CPU_PERFORMANCE, /* Select allocators in order from fastest to slowest on the CPU */
+ ALLOCATOR_ORDER_CPU_GPU_PERFORMANCE, /* Select allocators in order from fastest to slowest on the CPU and GPU */
+
+ ALLOCATOR_ORDER_COUNT
+} kbase_phys_allocator_order;
+
+
+/* A simple structure to keep a sorted list of
+ * osk_phy_allocator pointers.
+ * Used by the iterator object
+ */
+typedef struct kbase_phys_allocator_array
+{
+ /* the allocators */
+ osk_phy_allocator * allocs;
+ osk_phy_allocator ** sorted_allocs[ALLOCATOR_ORDER_COUNT];
+ /* number of allocators */
+ unsigned int count;
+
+#if MALI_DEBUG
+ mali_bool it_bound;
+#endif /* MALI_DEBUG */
+} kbase_phys_allocator_array;
+
+/**
+ * Instrumentation State Machine States:
+ * DISABLED - requires instrumentation to be enabled
+ * IDLE - state machine is active and ready for a command.
+ * DUMPING - hardware is currently dumping a frame.
+ * POSTCLEANING- hardware is currently cleaning and invalidating caches.
+ * PRECLEANING - same as POSTCLEANING, except on completion, state machine will transiton to CLEANED instead of IDLE.
+ * CLEANED - cache clean completed, waiting for Instrumentation setup.
+ * ERROR - an error has occured during DUMPING (page fault).
+ */
+
+typedef enum
+{
+ KBASE_INSTR_STATE_DISABLED = 0,
+ KBASE_INSTR_STATE_IDLE,
+ KBASE_INSTR_STATE_DUMPING,
+ KBASE_INSTR_STATE_CLEANED,
+ KBASE_INSTR_STATE_PRECLEANING,
+ KBASE_INSTR_STATE_POSTCLEANING,
+ KBASE_INSTR_STATE_RESETTING,
+ KBASE_INSTR_STATE_FAULT
+
+} kbase_instr_state;
+
+
+typedef struct kbasep_mem_device
+{
+#if MALI_USE_UMP == 1
+ u32 ump_device_id; /* Which UMP device this GPU should be mapped to.
+ Read-only, copied from platform configuration on startup.*/
+#endif /* MALI_USE_UMP == 1 */
+
+ u32 per_process_memory_limit; /* How much memory (in bytes) a single process can access.
+ Read-only, copied from platform configuration on startup. */
+ kbasep_mem_usage usage; /* Tracks usage of OS shared memory. Initialized with platform
+ configuration data, updated when OS memory is allocated/freed.*/
+ kbase_phys_allocator_array allocators; /* List of available physical memory allocators */
+} kbasep_mem_device;
+
+
+#define KBASE_TRACE_CODE( X ) KBASE_TRACE_CODE_ ## X
+
+typedef enum
+{
+ /* IMPORTANT: USE OF SPECIAL #INCLUDE OF NON-STANDARD HEADER FILE
+ * THIS MUST BE USED AT THE START OF THE ENUM */
+#define KBASE_TRACE_CODE_MAKE_CODE( X ) KBASE_TRACE_CODE( X )
+#include "mali_kbase_trace_defs.h"
+#undef KBASE_TRACE_CODE_MAKE_CODE
+ /* Comma on its own, to extend the list */
+ ,
+ /* Must be the last in the enum */
+ KBASE_TRACE_CODE_COUNT
+} kbase_trace_code;
+
+#define KBASE_TRACE_FLAG_REFCOUNT (((u8)1) << 0)
+#define KBASE_TRACE_FLAG_JOBSLOT (((u8)1) << 1)
+
+typedef struct kbase_trace
+{
+ osk_timeval timestamp;
+ u32 thread_id;
+ u32 cpu;
+ void *ctx;
+ void *uatom;
+ u64 gpu_addr;
+ u32 info_val;
+ u8 code;
+ u8 jobslot;
+ u8 refcount;
+ u8 flags;
+} kbase_trace;
+
+struct kbase_device {
+ const kbase_device_info *dev_info;
+ kbase_jm_slot jm_slots[BASE_JM_MAX_NR_SLOTS];
+ s8 slot_submit_count_irq[BASE_JM_MAX_NR_SLOTS];
+ kbase_os_device osdev;
+ kbase_pm_device_data pm;
+ kbasep_js_device_data js_data;
+ kbasep_mem_device memdev;
+
+ kbase_as as[BASE_MAX_NR_AS];
+
+ osk_phy_allocator mmu_fault_allocator;
+ osk_phy_addr mmu_fault_pages[4];
+ osk_spinlock_irq mmu_mask_change;
+
+ kbase_gpu_props gpu_props;
+
+ /**< List of SW workarounds for HW issues */
+ unsigned long hw_issues_mask[(BASE_HW_ISSUE_END + OSK_BITS_PER_LONG - 1)/OSK_BITS_PER_LONG];
+
+ /* Cached present bitmaps - these are the same as the corresponding hardware registers */
+ u64 shader_present_bitmap;
+ u64 tiler_present_bitmap;
+ u64 l2_present_bitmap;
+ u64 l3_present_bitmap;
+
+ /* Bitmaps of cores that are currently in use (running jobs).
+ * These should be kept up to date by the job scheduler.
+ *
+ * pm.power_change_lock should be held when accessing these members.
+ *
+ * kbase_pm_check_transitions should be called when bits are cleared to
+ * update the power management system and allow transitions to occur. */
+ u64 shader_inuse_bitmap;
+ u64 tiler_inuse_bitmap;
+
+ /* Refcount for cores in use */
+ u32 shader_inuse_cnt[64];
+ u32 tiler_inuse_cnt[64];
+
+ /* Bitmaps of cores the JS needs for jobs ready to run */
+ u64 shader_needed_bitmap;
+ u64 tiler_needed_bitmap;
+
+ /* Refcount for cores needed */
+ u32 shader_needed_cnt[64];
+ u32 tiler_needed_cnt[64];
+
+ /* Bitmaps of cores that are currently available (powered up and the power policy is happy for jobs to be
+ * submitted to these cores. These are updated by the power management code. The job scheduler should avoid
+ * submitting new jobs to any cores that are not marked as available.
+ *
+ * pm.power_change_lock should be held when accessing these members.
+ */
+ u64 shader_available_bitmap;
+ u64 tiler_available_bitmap;
+
+ s8 nr_hw_address_spaces; /**< Number of address spaces in the GPU (constant after driver initialisation) */
+ s8 nr_user_address_spaces; /**< Number of address spaces available to user contexts */
+
+ /* Structure used for instrumentation and HW counters dumping */
+ struct {
+ /* The lock should be used when accessing any of the following members */
+ osk_spinlock_irq lock;
+
+ kbase_context *kctx;
+ u64 addr;
+ osk_waitq waitqueue;
+ kbase_instr_state state;
+ } hwcnt;
+
+ /* Set when we're about to reset the GPU */
+ osk_atomic reset_gpu;
+#define KBASE_RESET_GPU_NOT_PENDING 0 /* The GPU reset isn't pending */
+#define KBASE_RESET_GPU_PREPARED 1 /* kbase_prepare_to_reset_gpu has been called */
+#define KBASE_RESET_GPU_COMMITTED 2 /* kbase_reset_gpu has been called - the reset will now definitely happen
+ * within the timeout period */
+#define KBASE_RESET_GPU_HAPPENING 3 /* The GPU reset process is currently occuring (timeout has expired or
+ * kbasep_try_reset_gpu_early was called) */
+
+ /* Work queue and work item for performing the reset in */
+ osk_workq reset_workq;
+ osk_workq_work reset_work;
+ /* Signalled when reset_gpu==KBASE_RESET_GPU_NOT_PENDING */
+ osk_waitq reset_waitq;
+ osk_timer reset_timer;
+
+ /*value to be written to the irq_throttle register each time an irq is served */
+ osk_atomic irq_throttle_cycles;
+
+ const kbase_attribute *config_attributes;
+
+ /* >> BASE_HW_ISSUE_8401 >> */
+#define KBASE_8401_WORKAROUND_COMPUTEJOB_COUNT 3
+ kbase_context *workaround_kctx;
+ osk_virt_addr workaround_compute_job_va[KBASE_8401_WORKAROUND_COMPUTEJOB_COUNT];
+ osk_phy_addr workaround_compute_job_pa[KBASE_8401_WORKAROUND_COMPUTEJOB_COUNT];
+ /* << BASE_HW_ISSUE_8401 << */
+
+#if KBASE_TRACE_ENABLE != 0
+ osk_spinlock_irq trace_lock;
+ u16 trace_first_out;
+ u16 trace_next_in;
+ kbase_trace *trace_rbuf;
+#endif
+
+#if MALI_CUSTOMER_RELEASE == 0
+ /* This is used to override the current job scheduler values for
+ * KBASE_CONFIG_ATTR_JS_STOP_STOP_TICKS_SS
+ * KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS
+ * KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS
+ * KBASE_CONFIG_ATTR_JS_RESET_TICKS_SS
+ * KBASE_CONFIG_ATTR_JS_RESET_TICKS_NSS.
+ *
+ * These values are set via the js_timeouts sysfs file.
+ */
+ u32 js_soft_stop_ticks;
+ u32 js_hard_stop_ticks_ss;
+ u32 js_hard_stop_ticks_nss;
+ u32 js_reset_ticks_ss;
+ u32 js_reset_ticks_nss;
+#endif
+ /* Platform specific private data to be accessed by mali_kbase_config_xxx.c only */
+ void *platform_context;
+};
+
+struct kbase_context
+{
+ kbase_device *kbdev;
+ osk_phy_allocator pgd_allocator;
+ osk_phy_addr pgd;
+ osk_dlist event_list;
+ osk_mutex event_mutex;
+ mali_bool event_closed;
+
+ u64 *mmu_teardown_pages;
+
+ osk_mutex reg_lock; /* To be converted to a rwlock? */
+ osk_dlist reg_list; /* Ordered list of GPU regions */
+
+ kbase_os_context osctx;
+ kbase_jd_context jctx;
+ kbasep_mem_usage usage;
+ ukk_session ukk_session;
+ u32 nr_outstanding_atoms;
+ osk_waitq complete_outstanding_waitq; /*if there are too many outstanding atoms
+ *per context we wait on this waitqueue
+ *to be signaled before submitting more jobs
+ */
+
+ /** This is effectively part of the Run Pool, because it only has a valid
+ * setting (!=KBASEP_AS_NR_INVALID) whilst the context is scheduled in
+ *
+ * The kbasep_js_device_data::runpool_irq::lock must be held whilst accessing
+ * this.
+ *
+ * If the context relating to this as_nr is required, you must use
+ * kbasep_js_runpool_retain_ctx() to ensure that the context doesn't disappear
+ * whilst you're using it. Alternatively, just hold the kbasep_js_device_data::runpool_irq::lock
+ * to ensure the context doesn't disappear (but this has restrictions on what other locks
+ * you can take whilst doing this) */
+ int as_nr;
+
+ /* NOTE:
+ *
+ * Flags are in jctx.sched_info.ctx.flags
+ * Mutable flags *must* be accessed under jctx.sched_info.ctx.jsctx_mutex
+ *
+ * All other flags must be added there */
+};
+
+typedef enum kbase_reg_access_type
+{
+ REG_READ,
+ REG_WRITE
+} kbase_reg_access_type;
+
+
+typedef enum kbase_share_attr_bits
+{
+ /* (1ULL << 8) bit is reserved */
+ SHARE_BOTH_BITS = (2ULL << 8), /* inner and outer shareable coherency */
+ SHARE_INNER_BITS = (3ULL << 8) /* inner shareable coherency */
+} kbase_share_attr_bits;
+
+
+#endif /* _KBASE_DEFS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_device.c
+ * Base kernel device APIs
+ */
+
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_kbase_defs.h>
+#include <kbase/src/common/mali_kbase_hw.h>
+
+/* NOTE: Magic - 0x45435254 (TRCE in ASCII).
+ * Supports tracing feature provided in the base module.
+ * Please keep it in sync with the value of base module.
+ */
+#define TRACE_BUFFER_HEADER_SPECIAL 0x45435254
+
+#ifdef MALI_PLATFORM_CONFIG_VEXPRESS
+#if (MALI_BACKEND_KERNEL && (!MALI_LICENSE_IS_GPL || MALI_FAKE_PLATFORM_DEVICE))
+extern kbase_attribute config_attributes_hw_issue_8408[];
+#endif /* (MALI_BACKEND_KERNEL && (!MALI_LICENSE_IS_GPL || MALI_FAKE_PLATFORM_DEVICE)) */
+#endif /* MALI_PLATFORM_CONFIG_VEXPRESS */
+
+/* This array is referenced at compile time, it cannot be made static... */
+const kbase_device_info kbase_dev_info[] = {
+ {
+ KBASE_MALI_T6XM,
+ (KBASE_FEATURE_HAS_MODEL_PMU)
+ },
+ {
+ KBASE_MALI_T6F1,
+ (KBASE_FEATURE_NEEDS_REG_DELAY |
+ KBASE_FEATURE_DELAYED_PERF_WRITE_STATUS |
+ KBASE_FEATURE_HAS_16BIT_PC)
+ },
+ {
+ KBASE_MALI_T601, 0
+ },
+ {
+ KBASE_MALI_T604, 0
+ },
+ {
+ KBASE_MALI_T608, 0
+ },
+};
+
+#if KBASE_TRACE_ENABLE != 0
+STATIC CONST char *kbasep_trace_code_string[] =
+{
+ /* IMPORTANT: USE OF SPECIAL #INCLUDE OF NON-STANDARD HEADER FILE
+ * THIS MUST BE USED AT THE START OF THE ARRAY */
+#define KBASE_TRACE_CODE_MAKE_CODE( X ) # X
+#include "mali_kbase_trace_defs.h"
+#undef KBASE_TRACE_CODE_MAKE_CODE
+};
+#endif
+
+STATIC mali_error kbasep_trace_init( kbase_device *kbdev );
+STATIC void kbasep_trace_term( kbase_device *kbdev );
+STATIC void kbasep_trace_hook_wrapper( void *param );
+
+void kbasep_as_do_poke(osk_workq_work * work);
+void kbasep_reset_timer_callback(void *data);
+void kbasep_reset_timeout_worker(osk_workq_work *data);
+
+kbase_device *kbase_device_alloc(void)
+{
+ return osk_calloc(sizeof(kbase_device));
+}
+
+mali_error kbase_device_init(kbase_device *kbdev, const kbase_device_info *dev_info)
+{
+ osk_error osk_err;
+ int i; /* i used after the for loop, don't reuse ! */
+
+ kbdev->dev_info = dev_info;
+
+ osk_err = osk_spinlock_irq_init(&kbdev->mmu_mask_change, OSK_LOCK_ORDER_MMU_MASK);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto fail;
+ }
+
+ /* Initialize platform specific context */
+ if(MALI_FALSE == kbasep_platform_device_init(kbdev))
+ {
+ goto free_mmu_lock;
+ }
+
+ /* Ensure we can access the GPU registers */
+ kbase_pm_register_access_enable(kbdev);
+
+ /* Get the list of workarounds for issues on the current HW (identified by the GPU_ID register) */
+ if (MALI_ERROR_NONE != kbase_hw_set_issues_mask(kbdev))
+ {
+ kbase_pm_register_access_disable(kbdev);
+ goto free_platform;
+ }
+
+ /* Find out GPU properties based on the GPU feature registers */
+ kbase_gpuprops_set(kbdev);
+
+ kbdev->nr_hw_address_spaces = kbdev->gpu_props.num_address_spaces;
+
+ /* We're done accessing the GPU registers for now. */
+ kbase_pm_register_access_disable(kbdev);
+
+ for (i = 0; i < kbdev->nr_hw_address_spaces; i++)
+ {
+ const char format[] = "mali_mmu%d";
+ char name[sizeof(format)];
+ const char poke_format[] = "mali_mmu%d_poker"; /* BASE_HW_ISSUE_8316 */
+ char poke_name[sizeof(poke_format)]; /* BASE_HW_ISSUE_8316 */
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8316))
+ {
+ if (0 > osk_snprintf(poke_name, sizeof(poke_name), poke_format, i))
+ {
+ goto free_workqs;
+ }
+ }
+
+ if (0 > osk_snprintf(name, sizeof(name), format, i))
+ {
+ goto free_workqs;
+ }
+
+ kbdev->as[i].number = i;
+ kbdev->as[i].fault_addr = 0ULL;
+ osk_err = osk_workq_init(&kbdev->as[i].pf_wq, name, 0);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto free_workqs;
+ }
+ osk_err = osk_mutex_init(&kbdev->as[i].transaction_mutex, OSK_LOCK_ORDER_AS);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ osk_workq_term(&kbdev->as[i].pf_wq);
+ goto free_workqs;
+ }
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8316))
+ {
+ osk_err = osk_workq_init(&kbdev->as[i].poke_wq, poke_name, 0);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ osk_workq_term(&kbdev->as[i].pf_wq);
+ osk_mutex_term(&kbdev->as[i].transaction_mutex);
+ goto free_workqs;
+ }
+ osk_workq_work_init(&kbdev->as[i].poke_work, kbasep_as_do_poke);
+ osk_err = osk_timer_init(&kbdev->as[i].poke_timer);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ osk_workq_term(&kbdev->as[i].poke_wq);
+ osk_workq_term(&kbdev->as[i].pf_wq);
+ osk_mutex_term(&kbdev->as[i].transaction_mutex);
+ goto free_workqs;
+ }
+ osk_timer_callback_set(&kbdev->as[i].poke_timer, kbasep_as_poke_timer_callback , &kbdev->as[i]);
+ osk_atomic_set(&kbdev->as[i].poke_refcount, 0);
+ }
+ }
+ /* don't change i after this point */
+
+ osk_err = osk_spinlock_irq_init(&kbdev->hwcnt.lock, OSK_LOCK_ORDER_HWCNT);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto free_workqs;
+ }
+
+ kbdev->hwcnt.state = KBASE_INSTR_STATE_DISABLED;
+ osk_err = osk_waitq_init(&kbdev->hwcnt.waitqueue);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto free_hwcnt_lock;
+ }
+
+ if (OSK_ERR_NONE != osk_workq_init(&kbdev->reset_workq, "Mali reset workqueue", 0))
+ {
+ goto free_hwcnt_waitq;
+ }
+
+ osk_workq_work_init(&kbdev->reset_work, kbasep_reset_timeout_worker);
+
+ osk_err = osk_waitq_init(&kbdev->reset_waitq);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto free_reset_workq;
+ }
+
+ osk_err = osk_timer_init(&kbdev->reset_timer);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto free_reset_waitq;
+ }
+ osk_timer_callback_set(&kbdev->reset_timer, kbasep_reset_timer_callback, kbdev);
+
+ if ( kbasep_trace_init( kbdev ) != MALI_ERROR_NONE )
+ {
+ goto free_reset_timer;
+ }
+
+ osk_debug_assert_register_hook( &kbasep_trace_hook_wrapper, kbdev );
+
+#ifdef MALI_PLATFORM_CONFIG_VEXPRESS
+#if (MALI_BACKEND_KERNEL && (!MALI_LICENSE_IS_GPL || MALI_FAKE_PLATFORM_DEVICE))
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8408))
+ {
+ /* BASE_HW_ISSUE_8408 requires a configuration with different timeouts for
+ * the vexpress platform */
+ kbdev->config_attributes = config_attributes_hw_issue_8408;
+ }
+#endif /* (MALI_BACKEND_KERNEL && (!MALI_LICENSE_IS_GPL || MALI_FAKE_PLATFORM_DEVICE)) */
+#endif /* MALI_PLATFORM_CONFIG=vexpress */
+
+ return MALI_ERROR_NONE;
+
+free_reset_timer:
+ osk_timer_term(&kbdev->reset_timer);
+free_reset_waitq:
+ osk_waitq_term(&kbdev->reset_waitq);
+free_reset_workq:
+ osk_workq_term(&kbdev->reset_workq);
+free_hwcnt_waitq:
+ osk_waitq_term(&kbdev->hwcnt.waitqueue);
+free_hwcnt_lock:
+ osk_spinlock_irq_term(&kbdev->hwcnt.lock);
+free_workqs:
+ while (i > 0)
+ {
+ i--;
+ osk_mutex_term(&kbdev->as[i].transaction_mutex);
+ osk_workq_term(&kbdev->as[i].pf_wq);
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8316))
+ {
+ osk_workq_term(&kbdev->as[i].poke_wq);
+ osk_timer_term(&kbdev->as[i].poke_timer);
+ }
+ }
+free_platform:
+ kbasep_platform_device_term(kbdev);
+free_mmu_lock:
+ osk_spinlock_irq_term(&kbdev->mmu_mask_change);
+fail:
+ return MALI_ERROR_FUNCTION_FAILED;
+}
+
+void kbase_device_term(kbase_device *kbdev)
+{
+ int i;
+
+ osk_debug_assert_register_hook( NULL, NULL );
+
+ kbasep_trace_term( kbdev );
+
+ osk_timer_term(&kbdev->reset_timer);
+ osk_waitq_term(&kbdev->reset_waitq);
+ osk_workq_term(&kbdev->reset_workq);
+
+ for (i = 0; i < kbdev->nr_hw_address_spaces; i++)
+ {
+ osk_mutex_term(&kbdev->as[i].transaction_mutex);
+ osk_workq_term(&kbdev->as[i].pf_wq);
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8316))
+ {
+ osk_timer_term(&kbdev->as[i].poke_timer);
+ osk_workq_term(&kbdev->as[i].poke_wq);
+ }
+ }
+
+ kbasep_platform_device_term(kbdev);
+
+ osk_spinlock_irq_term(&kbdev->hwcnt.lock);
+ osk_waitq_term(&kbdev->hwcnt.waitqueue);
+}
+
+void kbase_device_free(kbase_device *kbdev)
+{
+ osk_free(kbdev);
+}
+
+int kbase_device_has_feature(kbase_device *kbdev, u32 feature)
+{
+ return !!(kbdev->dev_info->features & feature);
+}
+KBASE_EXPORT_TEST_API(kbase_device_has_feature)
+
+kbase_midgard_type kbase_device_get_type(kbase_device *kbdev)
+{
+ return kbdev->dev_info->dev_type;
+}
+KBASE_EXPORT_TEST_API(kbase_device_get_type)
+
+void kbase_device_trace_buffer_install(kbase_context * kctx, u32 * tb, size_t size)
+{
+ OSK_ASSERT(kctx);
+ OSK_ASSERT(tb);
+
+ /* set up the header */
+ /* magic number in the first 4 bytes */
+ tb[0] = TRACE_BUFFER_HEADER_SPECIAL;
+ /* Store (write offset = 0, wrap counter = 0, transaction active = no)
+ * write offset 0 means never written.
+ * Offsets 1 to (wrap_offset - 1) used to store values when trace started
+ */
+ tb[1] = 0;
+
+ /* install trace buffer */
+ osk_spinlock_irq_lock(&kctx->jctx.tb_lock);
+ kctx->jctx.tb_wrap_offset = size / 8;
+ kctx->jctx.tb = tb;
+ osk_spinlock_irq_unlock(&kctx->jctx.tb_lock);
+}
+
+void kbase_device_trace_buffer_uninstall(kbase_context * kctx)
+{
+ OSK_ASSERT(kctx);
+ osk_spinlock_irq_lock(&kctx->jctx.tb_lock);
+ kctx->jctx.tb = NULL;
+ kctx->jctx.tb_wrap_offset = 0;
+ osk_spinlock_irq_unlock(&kctx->jctx.tb_lock);
+}
+
+void kbase_device_trace_register_access(kbase_context * kctx, kbase_reg_access_type type, u16 reg_offset, u32 reg_value)
+{
+ osk_spinlock_irq_lock(&kctx->jctx.tb_lock);
+ if (kctx->jctx.tb)
+ {
+ u16 wrap_count;
+ u16 write_offset;
+ osk_atomic dummy; /* osk_atomic_set called to use memory barriers until OSK get's them */
+ u32 * tb = kctx->jctx.tb;
+ u32 header_word;
+
+ header_word = tb[1];
+ OSK_ASSERT(0 == (header_word & 0x1));
+
+ wrap_count = (header_word >> 1) & 0x7FFF;
+ write_offset = (header_word >> 16) & 0xFFFF;
+
+ /* mark as transaction in progress */
+ tb[1] |= 0x1;
+ osk_atomic_set(&dummy, 1);
+
+ /* calculate new offset */
+ write_offset++;
+ if (write_offset == kctx->jctx.tb_wrap_offset)
+ {
+ /* wrap */
+ write_offset = 1;
+ wrap_count++;
+ wrap_count &= 0x7FFF; /* 15bit wrap counter */
+ }
+
+ /* store the trace entry at the selected offset */
+ tb[write_offset * 2 + 0] = (reg_offset & ~0x3) | ((type == REG_WRITE) ? 0x1 : 0x0);
+ tb[write_offset * 2 + 1] = reg_value;
+
+ osk_atomic_set(&dummy, 1);
+
+ /* new header word */
+ header_word = (write_offset << 16) | (wrap_count << 1) | 0x0; /* transaction complete */
+ tb[1] = header_word;
+ }
+ osk_spinlock_irq_unlock(&kctx->jctx.tb_lock);
+}
+
+void kbase_reg_write(kbase_device *kbdev, u16 offset, u32 value, kbase_context * kctx)
+{
+ OSK_ASSERT(kbdev->pm.gpu_powered);
+ OSK_ASSERT(kctx==NULL || kctx->as_nr != KBASEP_AS_NR_INVALID);
+ OSK_PRINT_INFO(OSK_BASE_CORE, "w: reg %04x val %08x", offset, value);
+ kbase_os_reg_write(kbdev, offset, value);
+ if (kctx && kctx->jctx.tb) kbase_device_trace_register_access(kctx, REG_WRITE, offset, value);
+}
+KBASE_EXPORT_TEST_API(kbase_reg_write)
+
+u32 kbase_reg_read(kbase_device *kbdev, u16 offset, kbase_context * kctx)
+{
+ u32 val;
+ OSK_ASSERT(kbdev->pm.gpu_powered);
+ OSK_ASSERT(kctx==NULL || kctx->as_nr != KBASEP_AS_NR_INVALID);
+ val = kbase_os_reg_read(kbdev, offset);
+ OSK_PRINT_INFO(OSK_BASE_CORE, "r: reg %04x val %08x", offset, val);
+ if (kctx && kctx->jctx.tb) kbase_device_trace_register_access(kctx, REG_READ, offset, val);
+ return val;
+}
+KBASE_EXPORT_TEST_API(kbase_reg_read)
+
+void kbase_report_gpu_fault(kbase_device *kbdev, int multiple)
+{
+ u32 status;
+ u64 address;
+
+ status = kbase_reg_read(kbdev, GPU_CONTROL_REG(GPU_FAULTSTATUS), NULL);
+ address = (u64)kbase_reg_read(kbdev, GPU_CONTROL_REG(GPU_FAULTADDRESS_HI), NULL) << 32;
+ address |= kbase_reg_read(kbdev, GPU_CONTROL_REG(GPU_FAULTADDRESS_LO), NULL);
+
+ OSK_PRINT_WARN(OSK_BASE_CORE, "GPU Fault 0x08%x (%s) at 0x%016llx", status, kbase_exception_name(status), address);
+ if (multiple)
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "There were multiple GPU faults - some have not been reported\n");
+ }
+}
+
+void kbase_gpu_interrupt(kbase_device * kbdev, u32 val)
+{
+ if (val & GPU_FAULT)
+ {
+ kbase_report_gpu_fault(kbdev, val & MULTIPLE_GPU_FAULTS);
+ }
+
+ if (val & RESET_COMPLETED)
+ {
+ kbase_pm_reset_done(kbdev);
+ }
+
+ if (val & PRFCNT_SAMPLE_COMPLETED)
+ {
+ kbase_instr_hwcnt_sample_done(kbdev);
+ }
+
+ if (val & CLEAN_CACHES_COMPLETED)
+ {
+ kbase_clean_caches_done(kbdev);
+ }
+
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_IRQ_CLEAR), val, NULL);
+
+ /* kbase_pm_check_transitions must be called after the IRQ has been cleared. This is because it might trigger
+ * further power transitions and we don't want to miss the interrupt raised to notify us that these further
+ * transitions have finished.
+ */
+ if (val & POWER_CHANGED_ALL)
+ {
+ kbase_pm_check_transitions(kbdev);
+ }
+}
+
+
+/*
+ * Device trace functions
+ */
+#if KBASE_TRACE_ENABLE != 0
+
+STATIC mali_error kbasep_trace_init( kbase_device *kbdev )
+{
+ osk_error osk_err;
+
+ void *rbuf = osk_malloc(sizeof(kbase_trace)*KBASE_TRACE_SIZE);
+
+ kbdev->trace_rbuf = rbuf;
+ osk_err = osk_spinlock_irq_init(&kbdev->trace_lock, OSK_LOCK_ORDER_TRACE);
+
+ if (rbuf == NULL || OSK_ERR_NONE != osk_err)
+ {
+ if ( rbuf != NULL )
+ {
+ osk_free( rbuf );
+ }
+ if ( osk_err == OSK_ERR_NONE )
+ {
+ osk_spinlock_irq_term(&kbdev->trace_lock);
+ }
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ return MALI_ERROR_NONE;
+}
+
+STATIC void kbasep_trace_term( kbase_device *kbdev )
+{
+ osk_spinlock_irq_term(&kbdev->trace_lock);
+ osk_free( kbdev->trace_rbuf );
+}
+
+void kbasep_trace_dump_msg( kbase_trace *trace_msg )
+{
+ char buffer[OSK_DEBUG_MESSAGE_SIZE];
+ s32 written = 0;
+
+ /* Initial part of message */
+ written += MAX( osk_snprintf(buffer+written, MAX((int)OSK_DEBUG_MESSAGE_SIZE-written,0),
+ "%d.%.6d,%d,%d,%s,%p,%p,%.8llx,",
+ trace_msg->timestamp.tv_sec,
+ trace_msg->timestamp.tv_usec,
+ trace_msg->thread_id,
+ trace_msg->cpu,
+ kbasep_trace_code_string[trace_msg->code],
+ trace_msg->ctx,
+ trace_msg->uatom,
+ trace_msg->gpu_addr ), 0 );
+
+ /* NOTE: Could add function callbacks to handle different message types */
+ if ( (trace_msg->flags & KBASE_TRACE_FLAG_JOBSLOT) != MALI_FALSE )
+ {
+ /* Jobslot present */
+ written += MAX( osk_snprintf(buffer+written, MAX((int)OSK_DEBUG_MESSAGE_SIZE-written,0),
+ "%d", trace_msg->jobslot), 0 );
+ }
+ written += MAX( osk_snprintf(buffer+written, MAX((int)OSK_DEBUG_MESSAGE_SIZE-written,0),
+ ","), 0 );
+
+ if ( (trace_msg->flags & KBASE_TRACE_FLAG_REFCOUNT) != MALI_FALSE )
+ {
+ /* Refcount present */
+ written += MAX( osk_snprintf(buffer+written, MAX((int)OSK_DEBUG_MESSAGE_SIZE-written,0),
+ "%d", trace_msg->refcount), 0 );
+ }
+ written += MAX( osk_snprintf(buffer+written, MAX((int)OSK_DEBUG_MESSAGE_SIZE-written,0),
+ ",", trace_msg->jobslot), 0 );
+
+ /* Rest of message */
+ written += MAX( osk_snprintf(buffer+written, MAX((int)OSK_DEBUG_MESSAGE_SIZE-written,0),
+ "0x%.8x", trace_msg->info_val), 0 );
+
+ OSK_PRINT( OSK_BASE_CORE, "%s", buffer );
+}
+
+void kbasep_trace_add(kbase_device *kbdev, kbase_trace_code code, void *ctx, void *uatom, u64 gpu_addr,
+ u8 flags, int refcount, int jobslot, u32 info_val )
+{
+ kbase_trace *trace_msg;
+
+ osk_spinlock_irq_lock( &kbdev->trace_lock );
+
+ trace_msg = &kbdev->trace_rbuf[kbdev->trace_next_in];
+
+ /* Fill the message */
+ osk_debug_get_thread_info( &trace_msg->thread_id, &trace_msg->cpu );
+
+ osk_gettimeofday(&trace_msg->timestamp);
+
+ trace_msg->code = code;
+ trace_msg->ctx = ctx;
+ trace_msg->uatom = uatom;
+ trace_msg->gpu_addr = gpu_addr;
+ trace_msg->jobslot = jobslot;
+ trace_msg->refcount = MIN((unsigned int)refcount, 0xFF) ;
+ trace_msg->info_val = info_val;
+ trace_msg->flags = flags;
+
+ /* Update the ringbuffer indices */
+ kbdev->trace_next_in = (kbdev->trace_next_in + 1) & KBASE_TRACE_MASK;
+ if ( kbdev->trace_next_in == kbdev->trace_first_out )
+ {
+ kbdev->trace_first_out = (kbdev->trace_first_out + 1) & KBASE_TRACE_MASK;
+ }
+
+ /* Done */
+
+ osk_spinlock_irq_unlock( &kbdev->trace_lock );
+}
+
+void kbasep_trace_clear(kbase_device *kbdev)
+{
+ osk_spinlock_irq_lock( &kbdev->trace_lock );
+ kbdev->trace_first_out = kbdev->trace_next_in;
+ osk_spinlock_irq_unlock( &kbdev->trace_lock );
+}
+
+void kbasep_trace_dump(kbase_device *kbdev)
+{
+ u32 start;
+ u32 end;
+
+
+ OSK_PRINT( OSK_BASE_CORE, "Dumping trace:\nsecs,nthread,cpu,code,ctx,uatom,gpu_addr,jobslot,refcount,info_val");
+ osk_spinlock_irq_lock( &kbdev->trace_lock );
+ start = kbdev->trace_first_out;
+ end = kbdev->trace_next_in;
+
+ while (start != end)
+ {
+ kbase_trace *trace_msg = &kbdev->trace_rbuf[start];
+ kbasep_trace_dump_msg( trace_msg );
+
+ start = (start + 1) & KBASE_TRACE_MASK;
+ }
+ OSK_PRINT( OSK_BASE_CORE, "TRACE_END");
+
+ osk_spinlock_irq_unlock( &kbdev->trace_lock );
+
+ KBASE_TRACE_CLEAR(kbdev);
+}
+
+STATIC void kbasep_trace_hook_wrapper( void *param )
+{
+ kbase_device *kbdev = (kbase_device*)param;
+ kbasep_trace_dump( kbdev );
+}
+
+#else /* KBASE_TRACE_ENABLE != 0 */
+STATIC mali_error kbasep_trace_init( kbase_device *kbdev )
+{
+ CSTD_UNUSED(kbdev);
+ return MALI_ERROR_NONE;
+}
+
+STATIC void kbasep_trace_term( kbase_device *kbdev )
+{
+ CSTD_UNUSED(kbdev);
+}
+
+STATIC void kbasep_trace_hook_wrapper( void *param )
+{
+ CSTD_UNUSED(param);
+}
+
+void kbasep_trace_add(kbase_device *kbdev, kbase_trace_code code, void *ctx, void *uatom, u64 gpu_addr,
+ u8 flags, int refcount, int jobslot, u32 info_val )
+{
+ CSTD_UNUSED(kbdev);
+ CSTD_UNUSED(code);
+ CSTD_UNUSED(ctx);
+ CSTD_UNUSED(uatom);
+ CSTD_UNUSED(gpu_addr);
+ CSTD_UNUSED(flags);
+ CSTD_UNUSED(refcount);
+ CSTD_UNUSED(jobslot);
+ CSTD_UNUSED(info_val);
+}
+
+void kbasep_trace_clear(kbase_device *kbdev)
+{
+ CSTD_UNUSED(kbdev);
+}
+
+void kbasep_trace_dump(kbase_device *kbdev)
+{
+ CSTD_UNUSED(kbdev);
+}
+#endif /* KBASE_TRACE_ENABLE != 0 */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#include <kbase/src/common/mali_kbase.h>
+
+#ifdef __KERNEL__
+#define beenthere(f, a...) pr_debug("%s:" f, __func__, ##a)
+#else
+#define beenthere(f, a...) OSK_PRINT_INFO(OSK_BASE_EVENT, "%s:" f, __func__, ##a)
+#endif
+
+STATIC void *kbase_event_process(kbase_context *ctx,
+ kbase_event *event)
+{
+ void *data;
+ void *ptr = event;
+ kbasep_js_policy *js_policy;
+
+ /*
+ * We're in the right user context, do some post processing
+ * before returning to user-mode.
+ */
+
+ OSK_ASSERT(ctx != NULL);
+ OSK_ASSERT(event->event_code);
+ js_policy = &(ctx->kbdev->js_data.policy);
+
+ if ((event->event_code & BASE_JD_SW_EVENT_TYPE_MASK) == BASE_JD_SW_EVENT_JOB)
+ {
+ kbase_jd_atom *katom = (void *)event->data;
+ /* return the offset in the ring buffer... */
+ data = (void *)((uintptr_t)katom->user_atom - (uintptr_t)ctx->jctx.pool);
+
+ if (katom->core_req & BASE_JD_REQ_EXTERNAL_RESOURCES)
+ {
+ kbase_jd_post_external_resources(katom);
+ }
+ else
+ {
+ /* perform the sync operations only on successful jobs */
+ kbase_post_job_sync(ctx,
+ base_jd_get_atom_syncset(katom->user_atom, 0),
+ katom->nr_syncsets);
+ }
+
+ if ((katom->core_req & BASE_JD_REQ_SOFT_JOB) == 0)
+ {
+ kbasep_js_policy_term_job( js_policy, ctx, katom );
+ }
+
+ ptr = katom;
+ /* As the event is integral part of the katom, return
+ * immediatly... */
+ goto out;
+ }
+
+ if ((event->event_code & BASE_JD_SW_EVENT_TYPE_MASK) == BASE_JD_SW_EVENT_BAG)
+ {
+ ptr = CONTAINER_OF(event, kbase_jd_bag, event);
+ goto assign;
+ }
+
+assign:
+ data = (void *)event->data; /* recast to discard const) */
+out:
+ osk_free(ptr);
+ return data;
+}
+
+int kbase_event_pending(kbase_context *ctx)
+{
+ int ret;
+
+ OSK_ASSERT(ctx);
+
+ osk_mutex_lock(&ctx->event_mutex);
+ ret = (MALI_FALSE == OSK_DLIST_IS_EMPTY(&ctx->event_list)) || (MALI_TRUE == ctx->event_closed);
+ osk_mutex_unlock(&ctx->event_mutex);
+
+ return ret;
+}
+KBASE_EXPORT_TEST_API(kbase_event_pending)
+
+int kbase_event_dequeue(kbase_context *ctx, base_jd_event *uevent)
+{
+ kbase_event *event;
+
+ OSK_ASSERT(ctx);
+
+ osk_mutex_lock(&ctx->event_mutex);
+
+ if (OSK_DLIST_IS_EMPTY(&ctx->event_list))
+ {
+ if (ctx->event_closed)
+ {
+ /* generate the BASE_JD_EVENT_DRV_TERMINATED message on the fly */
+ osk_mutex_unlock(&ctx->event_mutex);
+ uevent->event_code = BASE_JD_EVENT_DRV_TERMINATED;
+ uevent->data = NULL;
+ beenthere("event system closed, returning BASE_JD_EVENT_DRV_TERMINATED(0x%X)\n", BASE_JD_EVENT_DRV_TERMINATED);
+ return 0;
+ }
+ else
+ {
+ osk_mutex_unlock(&ctx->event_mutex);
+ return -1;
+ }
+ }
+
+ /* normal event processing */
+ event = OSK_DLIST_POP_FRONT(&ctx->event_list, kbase_event, entry);
+
+ osk_mutex_unlock(&ctx->event_mutex);
+
+ beenthere("event dequeuing %p\n", (void*)event);
+ uevent->event_code = event->event_code;
+ uevent->data = kbase_event_process(ctx, event);
+
+ return 0;
+}
+KBASE_EXPORT_TEST_API(kbase_event_dequeue)
+
+void kbase_event_post(kbase_context *ctx,
+ kbase_event *event)
+{
+ beenthere("event queuing %p\n", event);
+
+ OSK_ASSERT(ctx);
+
+ osk_mutex_lock(&ctx->event_mutex);
+ OSK_DLIST_PUSH_BACK(&ctx->event_list, event,
+ kbase_event, entry);
+ osk_mutex_unlock(&ctx->event_mutex);
+
+ kbase_event_wakeup(ctx);
+}
+KBASE_EXPORT_TEST_API(kbase_event_post)
+
+void kbase_event_close(kbase_context * kctx)
+{
+ osk_mutex_lock(&kctx->event_mutex);
+ kctx->event_closed = MALI_TRUE;
+ osk_mutex_unlock(&kctx->event_mutex);
+ kbase_event_wakeup(kctx);
+}
+
+mali_error kbase_event_init(kbase_context *kctx)
+{
+ osk_error osk_err;
+
+ OSK_ASSERT(kctx);
+
+ OSK_DLIST_INIT(&kctx->event_list);
+ osk_err = osk_mutex_init(&kctx->event_mutex, OSK_LOCK_ORDER_QUEUE);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ kctx->event_closed = MALI_FALSE;
+ return MALI_ERROR_NONE;
+}
+KBASE_EXPORT_TEST_API(kbase_event_init)
+
+void kbase_event_cleanup(kbase_context *kctx)
+{
+ OSK_ASSERT(kctx);
+
+ osk_mutex_lock(&kctx->event_mutex);
+ while (!OSK_DLIST_IS_EMPTY(&kctx->event_list))
+ {
+ kbase_event *event;
+ event = OSK_DLIST_POP_FRONT(&kctx->event_list,
+ kbase_event, entry);
+ beenthere("event dropping %p\n", event);
+ osk_free(event);
+ }
+ osk_mutex_unlock(&kctx->event_mutex);
+ osk_mutex_term(&kctx->event_mutex);
+}
+KBASE_EXPORT_TEST_API(kbase_event_cleanup)
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef MALI_GATOR_SUPPORT
+#define MALI_GATOR_SUPPORT 0
+#endif
+
+#if MALI_GATOR_SUPPORT
+#define GATOR_MAKE_EVENT(type,number) (((type) << 24) | ((number) << 16))
+#define GATOR_JOB_SLOT_START 1
+#define GATOR_JOB_SLOT_STOP 2
+#define GATOR_JOB_SLOT_SOFT_STOPPED 3
+void kbase_trace_mali_job_slots_event(u32 event);
+void kbase_trace_mali_pm_status(u32 event, u64 value);
+void kbase_trace_mali_pm_power_off(u32 event, u64 value);
+void kbase_trace_mali_pm_power_on(u32 event, u64 value);
+void kbase_trace_mali_page_fault_insert_pages(int event, u32 value);
+void kbase_trace_mali_mmu_as_in_use(int event);
+void kbase_trace_mali_mmu_as_released(int event);
+void kbase_trace_mali_total_alloc_pages_change(long long int event);
+#endif
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_gpuprops.c
+ * Base kernel property query APIs
+ */
+
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_midg_regmap.h>
+#include <kbase/src/common/mali_kbase_gpuprops.h>
+
+/**
+ * @brief Extracts bits from a 32-bit bitfield.
+ * @hideinitializer
+ *
+ * @param[in] value The value from which to extract bits.
+ * @param[in] offset The first bit to extract (0 being the LSB).
+ * @param[in] size The number of bits to extract.
+ * @return Bits [@a offset, @a offset + @a size) from @a value.
+ *
+ * @pre offset + size <= 32.
+ */
+/* from mali_cdsb.h */
+#define KBASE_UBFX32(value, offset, size) \
+ (((u32)(value) >> (u32)(offset)) & (u32)((1ULL << (u32)(size)) - 1))
+
+mali_error kbase_gpuprops_uk_get_props(kbase_context *kctx, kbase_uk_gpuprops * kbase_props)
+{
+ OSK_ASSERT(NULL != kctx);
+ OSK_ASSERT(NULL != kbase_props);
+
+ OSK_MEMCPY(&kbase_props->props, &kctx->kbdev->gpu_props.props, sizeof(kbase_props->props));
+
+ return MALI_ERROR_NONE;
+}
+
+STATIC void kbase_gpuprops_dump_registers(kbase_device * kbdev, kbase_gpuprops_regdump * regdump)
+{
+ int i;
+
+ OSK_ASSERT(NULL != kbdev);
+ OSK_ASSERT(NULL != regdump);
+
+ /* Fill regdump with the content of the relevant registers */
+ regdump->gpu_id = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(GPU_ID));
+ regdump->l2_features = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(L2_FEATURES));
+ regdump->l3_features = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(L3_FEATURES));
+ regdump->tiler_features = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(TILER_FEATURES));
+ regdump->mem_features = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(MEM_FEATURES));
+ regdump->mmu_features = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(MMU_FEATURES));
+ regdump->as_present = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(AS_PRESENT));
+ regdump->js_present = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(JS_PRESENT));
+
+ for(i = 0; i < MIDG_MAX_JOB_SLOTS; i++)
+ {
+ regdump->js_features[i] = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(JS_FEATURES_REG(i)));
+ }
+
+ for(i = 0; i < BASE_GPU_NUM_TEXTURE_FEATURES_REGISTERS; i++)
+ {
+ regdump->texture_features[i] = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(TEXTURE_FEATURES_REG(i)));
+ }
+
+ regdump->shader_present_lo = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(SHADER_PRESENT_LO));
+ regdump->shader_present_hi = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(SHADER_PRESENT_HI));
+
+ regdump->tiler_present_lo = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(TILER_PRESENT_LO));
+ regdump->tiler_present_hi = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(TILER_PRESENT_HI));
+
+ regdump->l2_present_lo = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(L2_PRESENT_LO));
+ regdump->l2_present_hi = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(L2_PRESENT_HI));
+
+ regdump->l3_present_lo = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(L3_PRESENT_LO));
+ regdump->l3_present_hi = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(L3_PRESENT_HI));
+}
+
+STATIC void kbase_gpuprops_construct_coherent_groups(base_gpu_props * const props)
+{
+ struct mali_base_gpu_coherent_group *current_group;
+ u64 group_present;
+ u64 group_mask;
+ u64 first_set, first_set_prev;
+ u32 num_groups = 0;
+
+ OSK_ASSERT(NULL != props);
+
+ props->coherency_info.coherency = props->raw_props.mem_features;
+ props->coherency_info.num_core_groups = osk_count_set_bits64(props->raw_props.l2_present);
+
+ if (props->coherency_info.coherency & GROUPS_L3_COHERENT)
+ {
+ /* Group is l3 coherent */
+ group_present = props->raw_props.l3_present;
+ }
+ else if (props->coherency_info.coherency & GROUPS_L2_COHERENT)
+ {
+ /* Group is l2 coherent */
+ group_present = props->raw_props.l2_present;
+ }
+ else
+ {
+ /* Group is l1 coherent */
+ group_present = props->raw_props.shader_present;
+ }
+
+ /*
+ * The coherent group mask can be computed from the l2/l3 present
+ * register.
+ *
+ * For the coherent group n:
+ * group_mask[n] = (first_set[n] - 1) & ~(first_set[n-1] - 1)
+ * where first_set is group_present with only its nth set-bit kept
+ * (i.e. the position from where a new group starts).
+ *
+ * For instance if the groups are l2 coherent and l2_present=0x0..01111:
+ * The first mask is:
+ * group_mask[1] = (first_set[1] - 1) & ~(first_set[0] - 1)
+ * = (0x0..010 - 1) & ~(0x0..01 - 1)
+ * = 0x0..00f
+ * The second mask is:
+ * group_mask[2] = (first_set[2] - 1) & ~(first_set[1] - 1)
+ * = (0x0..100 - 1) & ~(0x0..010 - 1)
+ * = 0x0..0f0
+ * And so on until all the bits from group_present have been cleared
+ * (i.e. there is no group left).
+ */
+
+ current_group = props->coherency_info.group;
+ first_set = group_present & ~(group_present - 1);
+
+ while (group_present != 0 && num_groups < BASE_MAX_COHERENT_GROUPS)
+ {
+ group_present -= first_set; /* Clear the current group bit */
+ first_set_prev = first_set;
+
+ first_set = group_present & ~(group_present - 1);
+ group_mask = (first_set - 1) & ~(first_set_prev - 1);
+
+ /* Populate the coherent_group structure for each group */
+ current_group->core_mask = group_mask & props->raw_props.shader_present;
+ current_group->num_cores = osk_count_set_bits64(current_group->core_mask);
+
+ num_groups++;
+ current_group++;
+ }
+
+ if (group_present != 0)
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Too many coherent groups (keeping only %d groups).\n", BASE_MAX_COHERENT_GROUPS);
+ }
+
+ props->coherency_info.num_groups = num_groups;
+}
+
+/**
+ * @brief Get the GPU configuration
+ *
+ * Fill the base_gpu_props structure with values from the GPU configuration registers
+ *
+ * @param gpu_props The base_gpu_props structure
+ * @param kbdev The kbase_device structure for the device
+ */
+static void kbase_gpuprops_get_props(base_gpu_props * gpu_props, kbase_device * kbdev)
+{
+ kbase_gpuprops_regdump regdump;
+ int i;
+
+ OSK_ASSERT(NULL != kbdev);
+ OSK_ASSERT(NULL != gpu_props);
+
+ /* Dump relevant registers */
+ kbase_gpuprops_dump_registers(kbdev, ®dump);
+
+ /* Populate the base_gpu_props structure */
+ gpu_props->core_props.version_status = KBASE_UBFX32(regdump.gpu_id, 0U, 4);
+ gpu_props->core_props.minor_revision = KBASE_UBFX32(regdump.gpu_id, 4U, 8);
+ gpu_props->core_props.major_revision = KBASE_UBFX32(regdump.gpu_id, 12U, 4);
+ gpu_props->core_props.product_id = KBASE_UBFX32(regdump.gpu_id, 16U, 16);
+ gpu_props->core_props.log2_program_counter_size = KBASE_GPU_PC_SIZE_LOG2;
+ gpu_props->core_props.gpu_speed_mhz = KBASE_GPU_SPEED_MHZ;
+ gpu_props->core_props.gpu_available_memory_size = OSK_MEM_PAGES << OSK_PAGE_SHIFT;
+
+ for(i = 0; i < BASE_GPU_NUM_TEXTURE_FEATURES_REGISTERS; i++)
+ {
+ gpu_props->core_props.texture_features[i] = regdump.texture_features[i];
+ }
+
+ gpu_props->l2_props.log2_line_size = KBASE_UBFX32(regdump.l2_features, 0U, 8);
+ gpu_props->l2_props.log2_cache_size = KBASE_UBFX32(regdump.l2_features, 16U, 8);
+
+ gpu_props->l3_props.log2_line_size = KBASE_UBFX32(regdump.l3_features, 0U, 8);
+ gpu_props->l3_props.log2_cache_size = KBASE_UBFX32(regdump.l3_features, 16U, 8);
+
+ gpu_props->tiler_props.bin_size_bytes = 1 << KBASE_UBFX32(regdump.tiler_features, 0U, 6);
+ gpu_props->tiler_props.max_active_levels = KBASE_UBFX32(regdump.tiler_features, 8U, 4);
+
+ gpu_props->raw_props.gpu_id = regdump.gpu_id;
+ gpu_props->raw_props.tiler_features = regdump.tiler_features;
+ gpu_props->raw_props.mem_features = regdump.mem_features;
+ gpu_props->raw_props.mmu_features = regdump.mmu_features;
+ gpu_props->raw_props.l2_features = regdump.l2_features;
+ gpu_props->raw_props.l3_features = regdump.l3_features;
+
+ gpu_props->raw_props.as_present = regdump.as_present;
+ gpu_props->raw_props.js_present = regdump.js_present;
+ gpu_props->raw_props.shader_present = ((u64)regdump.shader_present_hi << 32) + regdump.shader_present_lo;
+ gpu_props->raw_props.tiler_present = ((u64)regdump.tiler_present_hi << 32) + regdump.tiler_present_lo;
+ gpu_props->raw_props.l2_present = ((u64)regdump.l2_present_hi << 32) + regdump.l2_present_lo;
+ gpu_props->raw_props.l3_present = ((u64)regdump.l3_present_hi << 32) + regdump.l3_present_lo;
+
+ for(i = 0; i < MIDG_MAX_JOB_SLOTS; i++)
+ {
+ gpu_props->raw_props.js_features[i] = regdump.js_features[i];
+ }
+
+ /* Initialize the coherent_group structure for each group */
+ kbase_gpuprops_construct_coherent_groups(gpu_props);
+}
+
+void kbase_gpuprops_set(kbase_device *kbdev)
+{
+ kbase_gpu_props *gpu_props;
+ struct midg_raw_gpu_props *raw;
+
+ OSK_ASSERT(NULL != kbdev);
+ gpu_props = &kbdev->gpu_props;
+ raw = &gpu_props->props.raw_props;
+
+ /* Initialize the base_gpu_props structure */
+ kbase_gpuprops_get_props(&gpu_props->props, kbdev);
+
+ /* Populate kbase-only fields */
+ gpu_props->l2_props.associativity = KBASE_UBFX32(raw->l2_features, 8U, 8);
+ gpu_props->l2_props.external_bus_width = KBASE_UBFX32(raw->l2_features, 24U, 8);
+
+ gpu_props->l3_props.associativity = KBASE_UBFX32(raw->l3_features, 8U, 8);
+ gpu_props->l3_props.external_bus_width = KBASE_UBFX32(raw->l3_features, 24U, 8);
+
+ gpu_props->mem.core_group = KBASE_UBFX32(raw->mem_features, 0U, 1);
+ gpu_props->mem.supergroup = KBASE_UBFX32(raw->mem_features, 1U, 1);
+
+ gpu_props->mmu.va_bits = KBASE_UBFX32(raw->mmu_features, 0U, 8);
+ gpu_props->mmu.pa_bits = KBASE_UBFX32(raw->mmu_features, 8U, 8);
+
+ gpu_props->num_cores = osk_count_set_bits64(raw->shader_present);
+ gpu_props->num_core_groups = osk_count_set_bits64(raw->l2_present);
+ gpu_props->num_supergroups = osk_count_set_bits64(raw->l3_present);
+ gpu_props->num_address_spaces = osk_count_set_bits(raw->as_present);
+ gpu_props->num_job_slots = osk_count_set_bits(raw->js_present);
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_gpuprops.h
+ * Base kernel property query APIs
+ */
+
+#ifndef _KBASE_GPUPROPS_H_
+#define _KBASE_GPUPROPS_H_
+
+#include "mali_kbase_gpuprops_types.h"
+
+/* Forward definition - see mali_kbase.h */
+struct kbase_device;
+struct kbase_context;
+
+/**
+ * @brief Set up Kbase GPU properties.
+ *
+ * Set up Kbase GPU properties with information from the GPU registers
+ *
+ * @param kbdev The kbase_device structure for the device
+ */
+void kbase_gpuprops_set(struct kbase_device *kbdev);
+
+/**
+ * @brief Provide GPU properties to userside through UKU call.
+ *
+ * Fill the kbase_uk_gpuprops with values from GPU configuration registers.
+ *
+ * @param kctx The kbase_context structure
+ * @param kbase_props A copy of the kbase_uk_gpuprops structure from userspace
+ *
+ * @return MALI_ERROR_NONE on success. Any other value indicates failure.
+ */
+mali_error kbase_gpuprops_uk_get_props(struct kbase_context *kctx, kbase_uk_gpuprops * kbase_props);
+
+
+#endif /* _KBASE_GPUPROPS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_gpuprops_types.h
+ * Base kernel property query APIs
+ */
+
+#ifndef _KBASE_GPUPROPS_TYPES_H_
+#define _KBASE_GPUPROPS_TYPES_H_
+
+#include <kbase/mali_base_kernel.h>
+
+#define KBASE_GPU_SPEED_MHZ 123
+#define KBASE_GPU_PC_SIZE_LOG2 16U
+
+typedef struct kbase_gpuprops_regdump
+{
+ u32 gpu_id;
+ u32 l2_features;
+ u32 l3_features;
+ u32 tiler_features;
+ u32 mem_features;
+ u32 mmu_features;
+ u32 as_present;
+ u32 js_present;
+
+ u32 js_features[MIDG_MAX_JOB_SLOTS];
+
+ u32 texture_features[BASE_GPU_NUM_TEXTURE_FEATURES_REGISTERS];
+
+ u32 shader_present_lo;
+ u32 shader_present_hi;
+
+ u32 tiler_present_lo;
+ u32 tiler_present_hi;
+
+ u32 l2_present_lo;
+ u32 l2_present_hi;
+
+ u32 l3_present_lo;
+ u32 l3_present_hi;
+}kbase_gpuprops_regdump;
+
+typedef struct kbase_gpu_cache_props
+{
+ u8 associativity;
+ u8 external_bus_width;
+}kbase_gpu_cache_props;
+
+typedef struct kbase_gpu_mem_props
+{
+ u8 core_group;
+ u8 supergroup;
+}kbase_gpu_mem_props;
+
+typedef struct kbase_gpu_mmu_props
+{
+ u8 va_bits;
+ u8 pa_bits;
+}kbase_gpu_mmu_props;
+
+typedef struct mali_kbase_gpu_props
+{
+ /* kernel-only properties */
+ u8 num_cores;
+ u8 num_core_groups;
+ u8 num_supergroups;
+ u8 num_address_spaces;
+ u8 num_job_slots;
+
+ kbase_gpu_cache_props l2_props;
+ kbase_gpu_cache_props l3_props;
+
+ kbase_gpu_mem_props mem;
+ kbase_gpu_mmu_props mmu;
+
+ /**
+ * Implementation specific irq throttle value (us), should be adjusted during integration.
+ */
+ u32 irq_throttle_time_us;
+
+ /* Properties shared with userspace */
+ base_gpu_props props;
+}kbase_gpu_props;
+
+
+
+#endif /* _KBASE_GPUPROPS_TYPES_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Run-time work-arounds helpers
+ */
+
+#include <kbase/mali_base_hwconfig.h>
+#include <kbase/src/common/mali_midg_regmap.h>
+#include "mali_kbase.h"
+#include "mali_kbase_hw.h"
+
+mali_error kbase_hw_set_issues_mask(kbase_device *kbdev)
+{
+ const base_hw_issue *issues;
+
+#if MALI_BACKEND_KERNEL || MALI_NO_MALI
+ u32 gpu_id = kbase_os_reg_read(kbdev, GPU_CONTROL_REG(GPU_ID));
+
+ switch (gpu_id)
+ {
+ case GPU_ID_MAKE(GPU_ID_PI_T60X, 0, 0, GPU_ID_S_15DEV0):
+ case GPU_ID_MAKE(GPU_ID_PI_T65X, 0, 0, GPU_ID_S_15DEV0):
+ issues = base_hw_issues_t60x_t65x_r0p0_15dev0;
+ break;
+ case GPU_ID_MAKE(GPU_ID_PI_T60X, 0, 0, GPU_ID_S_EAC):
+ case GPU_ID_MAKE(GPU_ID_PI_T65X, 0, 0, GPU_ID_S_EAC):
+ issues = base_hw_issues_t60x_t65x_r0p0_eac;
+ break;
+ case GPU_ID_MAKE(GPU_ID_PI_T65X, 0, 1, 0):
+ issues = base_hw_issues_t65x_r0p1;
+ break;
+ case GPU_ID_MAKE(GPU_ID_PI_T60X, 1, 0, 0):
+ case GPU_ID_MAKE(GPU_ID_PI_T65X, 1, 0, 0):
+ issues = base_hw_issues_t60x_t65x_r1p0;
+ break;
+ case GPU_ID_MAKE(GPU_ID_PI_T62X, 0, 0, 0):
+ issues = base_hw_issues_t62x_r0p0;
+ break;
+ case GPU_ID_MAKE(GPU_ID_PI_T67X, 0, 0, 0):
+ issues = base_hw_issues_t67x_r0p0;
+ break;
+ default:
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Unknown GPU ID %x", gpu_id);
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ OSK_PRINT_INFO(OSK_BASE_CORE, "GPU identified as 0x%04x r%dp%d status %d",
+ (gpu_id & GPU_ID_VERSION_PRODUCT_ID) >> GPU_ID_VERSION_PRODUCT_ID_SHIFT,
+ (gpu_id & GPU_ID_VERSION_MAJOR) >> GPU_ID_VERSION_MAJOR_SHIFT,
+ (gpu_id & GPU_ID_VERSION_MINOR) >> GPU_ID_VERSION_MINOR_SHIFT,
+ (gpu_id & GPU_ID_VERSION_STATUS) >> GPU_ID_VERSION_STATUS_SHIFT);
+#else
+ /* We can only know that the model is used at compile-time */
+ issues = base_hw_issues_model;
+#endif
+
+ for (; *issues != BASE_HW_ISSUE_END; issues++)
+ {
+ osk_bitarray_set_bit(*issues, &kbdev->hw_issues_mask[0]);
+ }
+
+ return MALI_ERROR_NONE;
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Run-time work-arounds helpers
+ */
+
+#ifndef _KBASE_HW_H_
+#define _KBASE_HW_H_
+
+#include <osk/mali_osk.h>
+#include "mali_kbase_defs.h"
+
+/**
+ * @brief Tell whether a work-around should be enabled
+ */
+#define kbase_hw_has_issue(kbdev, issue)\
+ osk_bitarray_test_bit(issue, &(kbdev)->hw_issues_mask[0])
+
+/**
+ * @brief Set the HW issues mask depending on the GPU ID
+ */
+mali_error kbase_hw_set_issues_mask(kbase_device *kbdev);
+
+#endif /* _KBASE_HW_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_instr.c
+ * Base kernel instrumentation APIs.
+ */
+
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_midg_regmap.h>
+
+/**
+ * @brief Issue Cache Clean & Invalidate command to hardware
+ */
+static void kbasep_instr_hwcnt_cacheclean(kbase_device *kbdev)
+{
+ u32 irq_mask;
+
+ OSK_ASSERT(NULL != kbdev);
+
+ /* Enable interrupt */
+ irq_mask = kbase_reg_read(kbdev,GPU_CONTROL_REG(GPU_IRQ_MASK),NULL);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_IRQ_MASK), irq_mask | CLEAN_CACHES_COMPLETED, NULL);
+ /* clean&invalidate the caches so we're sure the mmu tables for the dump buffer is valid */
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_COMMAND), GPU_COMMAND_CLEAN_INV_CACHES, NULL);
+}
+
+/**
+ * @brief Enable HW counters collection
+ *
+ * Note: will wait for a cache clean to complete
+ */
+mali_error kbase_instr_hwcnt_enable(kbase_context * kctx, kbase_uk_hwcnt_setup * setup)
+{
+ mali_error err = MALI_ERROR_FUNCTION_FAILED;
+ kbasep_js_device_data *js_devdata;
+ mali_bool access_allowed;
+ u32 irq_mask;
+ kbase_device *kbdev;
+
+ OSK_ASSERT(NULL != kctx);
+ kbdev = kctx->kbdev;
+ OSK_ASSERT(NULL != kbdev);
+ OSK_ASSERT(NULL != setup);
+
+ js_devdata = &kbdev->js_data;
+ OSK_ASSERT(NULL != js_devdata);
+
+ /* Determine if the calling task has access to this capability */
+ access_allowed = kbase_security_has_capability(kctx, KBASE_SEC_INSTR_HW_COUNTERS_COLLECT, KBASE_SEC_FLAG_NOAUDIT);
+ if (MALI_FALSE == access_allowed)
+ {
+ goto out;
+ }
+
+ if ((setup->dump_buffer == 0ULL) ||
+ (setup->dump_buffer & (2048-1)))
+ {
+ /* alignment failure */
+ goto out;
+ }
+
+
+ /* Mark the context as active so the GPU is kept turned on */
+ kbase_pm_context_active(kbdev);
+
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+
+ if (kbdev->hwcnt.state == KBASE_INSTR_STATE_RESETTING)
+ {
+ /* GPU is being reset*/
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+ osk_waitq_wait(&kbdev->hwcnt.waitqueue);
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+ }
+
+
+ if (kbdev->hwcnt.state != KBASE_INSTR_STATE_DISABLED)
+ {
+ /* Instrumentation is already enabled */
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+ kbase_pm_context_idle(kbdev);
+ goto out;
+ }
+
+ /* Enable interrupt */
+ irq_mask = kbase_reg_read(kbdev,GPU_CONTROL_REG(GPU_IRQ_MASK),NULL);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_IRQ_MASK), irq_mask | PRFCNT_SAMPLE_COMPLETED, NULL);
+
+ /* In use, this context is the owner */
+ kbdev->hwcnt.kctx = kctx;
+ /* Remember the dump address so we can reprogram it later */
+ kbdev->hwcnt.addr = setup->dump_buffer;
+
+ /* Precleaning so that state does not transition to IDLE */
+ kbdev->hwcnt.state = KBASE_INSTR_STATE_PRECLEANING;
+ osk_waitq_clear(&kbdev->hwcnt.waitqueue);
+
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+
+ /* Clean&invalidate the caches so we're sure the mmu tables for the dump buffer is valid */
+ kbasep_instr_hwcnt_cacheclean(kbdev);
+ /* Wait for cacheclean to complete */
+ osk_waitq_wait(&kbdev->hwcnt.waitqueue);
+ OSK_ASSERT(kbdev->hwcnt.state == KBASE_INSTR_STATE_CLEANED);
+
+ /* Schedule the context in */
+ kbasep_js_schedule_privileged_ctx(kbdev, kctx);
+
+ /* Configure */
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_BASE_LO), setup->dump_buffer & 0xFFFFFFFF, kctx);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_BASE_HI), setup->dump_buffer >> 32, kctx);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_JM_EN), setup->jm_bm, kctx);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_SHADER_EN), setup->shader_bm, kctx);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_L3_CACHE_EN), setup->l3_cache_bm, kctx);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_MMU_L2_EN), setup->mmu_l2_bm, kctx);
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8186))
+ {
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_TILER_EN), 0, kctx);
+ }
+
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_CONFIG), (kctx->as_nr << PRFCNT_CONFIG_AS_SHIFT) | PRFCNT_CONFIG_MODE_MANUAL, kctx);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_TILER_EN), setup->tiler_bm, kctx);
+
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+
+ if (kbdev->hwcnt.state == KBASE_INSTR_STATE_RESETTING)
+ {
+ /* GPU is being reset*/
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+ osk_waitq_wait(&kbdev->hwcnt.waitqueue);
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+ }
+
+ kbdev->hwcnt.state = KBASE_INSTR_STATE_IDLE;
+ osk_waitq_set(&kbdev->hwcnt.waitqueue);
+
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+
+ err = MALI_ERROR_NONE;
+
+ OSK_PRINT_INFO( OSK_BASE_CORE, "HW counters dumping set-up for context %p", kctx);
+
+out:
+ return err;
+}
+KBASE_EXPORT_SYMBOL(kbase_instr_hwcnt_enable)
+
+/**
+ * @brief Disable HW counters collection
+ *
+ * Note: might sleep, waiting for an ongoing dump to complete
+ */
+mali_error kbase_instr_hwcnt_disable(kbase_context * kctx)
+{
+ mali_error err = MALI_ERROR_FUNCTION_FAILED;
+ u32 irq_mask;
+ kbase_device *kbdev;
+
+ OSK_ASSERT(NULL != kctx);
+ kbdev = kctx->kbdev;
+ OSK_ASSERT(NULL != kbdev);
+
+ while (1)
+ {
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+
+ if (kbdev->hwcnt.state == KBASE_INSTR_STATE_DISABLED)
+ {
+ /* Instrumentation is not enabled */
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+ goto out;
+ }
+
+ if (kbdev->hwcnt.kctx != kctx)
+ {
+ /* Instrumentation has been setup for another context */
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+ goto out;
+ }
+
+ if (kbdev->hwcnt.state == KBASE_INSTR_STATE_IDLE)
+ {
+ break;
+ }
+
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+
+ /* Ongoing dump/setup - wait for its completion */
+ osk_waitq_wait(&kbdev->hwcnt.waitqueue);
+ }
+
+ kbdev->hwcnt.state = KBASE_INSTR_STATE_DISABLED;
+ osk_waitq_clear(&kbdev->hwcnt.waitqueue);
+
+ /* Disable interrupt */
+ irq_mask = kbase_reg_read(kbdev,GPU_CONTROL_REG(GPU_IRQ_MASK),NULL);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_IRQ_MASK), irq_mask & ~PRFCNT_SAMPLE_COMPLETED, NULL);
+
+ /* Disable the counters */
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_CONFIG), 0, kctx);
+
+ kbdev->hwcnt.kctx = NULL;
+ kbdev->hwcnt.addr = 0ULL;
+
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+
+ /* Release the context, this implicitly (and indirectly) calls kbase_pm_context_idle */
+ kbasep_js_release_privileged_ctx(kbdev, kctx);
+
+ OSK_PRINT_INFO( OSK_BASE_CORE, "HW counters dumping disabled for context %p", kctx);
+
+ err = MALI_ERROR_NONE;
+
+out:
+ return err;
+}
+KBASE_EXPORT_SYMBOL(kbase_instr_hwcnt_disable)
+
+/**
+ * @brief Configure HW counters collection
+ */
+mali_error kbase_instr_hwcnt_setup(kbase_context * kctx, kbase_uk_hwcnt_setup * setup)
+{
+ mali_error err = MALI_ERROR_FUNCTION_FAILED;
+ kbase_device *kbdev;
+
+ OSK_ASSERT(NULL != kctx);
+
+ kbdev = kctx->kbdev;
+ OSK_ASSERT(NULL != kbdev);
+
+ if (NULL == setup)
+ {
+ /* Bad parameter - abort */
+ goto out;
+ }
+
+ if (setup->dump_buffer != 0ULL)
+ {
+ /* Enable HW counters */
+ err = kbase_instr_hwcnt_enable(kctx, setup);
+ }
+ else
+ {
+ /* Disable HW counters */
+ err = kbase_instr_hwcnt_disable(kctx);
+ }
+
+out:
+ return err;
+}
+
+/**
+ * @brief Issue Dump command to hardware
+ */
+mali_error kbase_instr_hwcnt_dump_irq(kbase_context * kctx)
+{
+ mali_error err = MALI_ERROR_FUNCTION_FAILED;
+ kbase_device *kbdev;
+
+ OSK_ASSERT(NULL != kctx);
+ kbdev = kctx->kbdev;
+ OSK_ASSERT(NULL != kbdev);
+
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+
+ OSK_ASSERT(kbdev->hwcnt.state != KBASE_INSTR_STATE_RESETTING);
+
+ if (kbdev->hwcnt.kctx != kctx)
+ {
+ /* The instrumentation has been setup for another context */
+ goto unlock;
+ }
+
+ if (kbdev->hwcnt.state != KBASE_INSTR_STATE_IDLE)
+ {
+ /* HW counters are disabled or another dump is ongoing */
+ goto unlock;
+ }
+
+ osk_waitq_clear(&kbdev->hwcnt.waitqueue);
+
+ /* Mark that we're dumping - the PF handler can signal that we faulted */
+ kbdev->hwcnt.state = KBASE_INSTR_STATE_DUMPING;
+
+ /* Reconfigure the dump address */
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_BASE_LO), kbdev->hwcnt.addr & 0xFFFFFFFF, NULL);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_BASE_HI), kbdev->hwcnt.addr >> 32, NULL);
+
+ /* Start dumping */
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_COMMAND), GPU_COMMAND_PRFCNT_SAMPLE, kctx);
+
+ OSK_PRINT_INFO( OSK_BASE_CORE, "HW counters dumping done for context %p", kctx);
+
+ err = MALI_ERROR_NONE;
+
+unlock:
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+ return err;
+}
+KBASE_EXPORT_SYMBOL(kbase_instr_hwcnt_dump_irq)
+
+/**
+ * @brief Tell whether the HW counters dump has completed
+ *
+ * Notes:
+ * - does not sleep
+ * - success will be set to MALI_TRUE if the dump succeeded or
+ * MALI_FALSE on failure
+ */
+mali_bool kbase_instr_hwcnt_dump_complete(kbase_context * kctx, mali_bool *success)
+{
+ mali_bool complete = MALI_FALSE;
+ kbase_device *kbdev;
+
+ OSK_ASSERT(NULL != kctx);
+ kbdev = kctx->kbdev;
+ OSK_ASSERT(NULL != kbdev);
+ OSK_ASSERT(NULL != success);
+
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+
+ if (kbdev->hwcnt.state == KBASE_INSTR_STATE_IDLE)
+ {
+ *success = MALI_TRUE;
+ complete = MALI_TRUE;
+ }
+ else if (kbdev->hwcnt.state == KBASE_INSTR_STATE_FAULT)
+ {
+ *success = MALI_FALSE;
+ complete = MALI_TRUE;
+ kbdev->hwcnt.state = KBASE_INSTR_STATE_IDLE;
+ }
+
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+
+ return complete;
+}
+KBASE_EXPORT_SYMBOL(kbase_instr_hwcnt_dump_complete)
+
+/**
+ * @brief Issue Dump command to hardware and wait for completion
+ */
+mali_error kbase_instr_hwcnt_dump(kbase_context * kctx)
+{
+ mali_error err = MALI_ERROR_FUNCTION_FAILED;
+ kbase_device *kbdev;
+
+ OSK_ASSERT(NULL != kctx);
+ kbdev = kctx->kbdev;
+ OSK_ASSERT(NULL != kbdev);
+
+ err = kbase_instr_hwcnt_dump_irq(kctx);
+ if (MALI_ERROR_NONE != err)
+ {
+ /* Can't dump HW counters */
+ goto out;
+ }
+
+ /* Wait for dump & cacheclean to complete */
+ osk_waitq_wait(&kbdev->hwcnt.waitqueue);
+
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+
+ if (kbdev->hwcnt.state == KBASE_INSTR_STATE_RESETTING)
+ {
+ /* GPU is being reset*/
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+ osk_waitq_wait(&kbdev->hwcnt.waitqueue);
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+ }
+
+ if (kbdev->hwcnt.state == KBASE_INSTR_STATE_FAULT)
+ {
+ err = MALI_ERROR_FUNCTION_FAILED;
+ kbdev->hwcnt.state = KBASE_INSTR_STATE_IDLE;
+ }
+ else
+ {
+ /* Dump done */
+ OSK_ASSERT(kbdev->hwcnt.state == KBASE_INSTR_STATE_IDLE);
+ err = MALI_ERROR_NONE;
+ }
+
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+out:
+ return err;
+}
+KBASE_EXPORT_SYMBOL(kbase_instr_hwcnt_dump)
+
+/**
+ * @brief Clear the HW counters
+ */
+mali_error kbase_instr_hwcnt_clear(kbase_context * kctx)
+{
+ mali_error err = MALI_ERROR_FUNCTION_FAILED;
+ kbase_device *kbdev;
+
+ OSK_ASSERT(NULL != kctx);
+ kbdev = kctx->kbdev;
+ OSK_ASSERT(NULL != kbdev);
+
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+
+ if (kbdev->hwcnt.state == KBASE_INSTR_STATE_RESETTING)
+ {
+ /* GPU is being reset*/
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+ osk_waitq_wait(&kbdev->hwcnt.waitqueue);
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+ }
+
+ /* Check it's the context previously set up and we're not already dumping */
+ if (kbdev->hwcnt.kctx != kctx ||
+ kbdev->hwcnt.state != KBASE_INSTR_STATE_IDLE)
+ {
+ goto out;
+ }
+
+ /* Clear the counters */
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_COMMAND), GPU_COMMAND_PRFCNT_CLEAR, kctx);
+
+ err = MALI_ERROR_NONE;
+
+out:
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+ return err;
+}
+KBASE_EXPORT_SYMBOL(kbase_instr_hwcnt_clear)
+
+/**
+ * @brief Dump complete interrupt received
+ */
+void kbase_instr_hwcnt_sample_done(kbase_device *kbdev)
+{
+ if (kbdev->hwcnt.state == KBASE_INSTR_STATE_FAULT)
+ {
+ osk_waitq_set(&kbdev->hwcnt.waitqueue);
+ }
+ else
+ {
+ /* Always clean and invalidate the cache after a successful dump */
+ kbdev->hwcnt.state = KBASE_INSTR_STATE_POSTCLEANING;
+ kbasep_instr_hwcnt_cacheclean(kbdev);
+ }
+}
+
+/**
+ * @brief Cache clean interrupt received
+ */
+void kbase_clean_caches_done(kbase_device *kbdev)
+{
+ u32 irq_mask;
+
+ if (kbdev->hwcnt.state != KBASE_INSTR_STATE_DISABLED)
+ {
+ /* Disable interrupt */
+ irq_mask = kbase_reg_read(kbdev,GPU_CONTROL_REG(GPU_IRQ_MASK),NULL);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_IRQ_MASK), irq_mask & ~CLEAN_CACHES_COMPLETED, NULL);
+
+ if (kbdev->hwcnt.state == KBASE_INSTR_STATE_PRECLEANING)
+ {
+ /* Don't return IDLE as we need kbase_instr_hwcnt_setup to continue rather than
+ allow access to another waiting thread */
+ kbdev->hwcnt.state = KBASE_INSTR_STATE_CLEANED;
+ }
+ else
+ {
+ /* All finished and idle */
+ kbdev->hwcnt.state = KBASE_INSTR_STATE_IDLE;
+ }
+
+ osk_waitq_set(&kbdev->hwcnt.waitqueue);
+ }
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+#ifdef CONFIG_DMA_SHARED_BUFFER
+#include <linux/dma-buf.h>
+#endif /* CONFIG_DMA_SHARED_BUFFER */
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_kbase_uku.h>
+#include <kbase/src/common/mali_kbase_js_affinity.h>
+#if MALI_USE_UMP == 1
+#include <ump/ump_kernel_interface.h>
+#endif /* MALI_USE_UMP == 1 */
+
+#define beenthere(f, a...) OSK_PRINT_INFO(OSK_BASE_JD, "%s:" f, __func__, ##a)
+
+/*
+ * This is the kernel side of the API. Only entry points are:
+ * - kbase_jd_submit(): Called from userspace to submit a single bag
+ * - kbase_jd_done(): Called from interrupt context to track the
+ * completion of a job.
+ * Callouts:
+ * - to the job manager (enqueue a job)
+ * - to the event subsystem (signals the completion/failure of bag/job-chains).
+ */
+
+STATIC INLINE void dep_raise_sem(u32 *sem, u8 dep)
+{
+ if (!dep)
+ return;
+
+ sem[BASEP_JD_SEM_WORD_NR(dep)] |= BASEP_JD_SEM_MASK_IN_WORD(dep);
+}
+KBASE_EXPORT_TEST_API(dep_raise_sem)
+
+STATIC INLINE void dep_clear_sem(u32 *sem, u8 dep)
+{
+ if (!dep)
+ return;
+
+ sem[BASEP_JD_SEM_WORD_NR(dep)] &= ~BASEP_JD_SEM_MASK_IN_WORD(dep);
+}
+KBASE_EXPORT_TEST_API(dep_clear_sem)
+
+STATIC INLINE int dep_get_sem(u32 *sem, u8 dep)
+{
+ if (!dep)
+ return 0;
+
+ return !!(sem[BASEP_JD_SEM_WORD_NR(dep)] & BASEP_JD_SEM_MASK_IN_WORD(dep));
+}
+KBASE_EXPORT_TEST_API(dep_get_sem)
+
+STATIC INLINE mali_bool jd_add_dep(kbase_jd_context *ctx,
+ kbase_jd_atom *katom, u8 d)
+{
+ kbase_jd_dep_queue *dq = &ctx->dep_queue;
+ u8 s = katom->pre_dep.dep[d];
+
+ if (!dep_get_sem(ctx->dep_queue.sem, s))
+ return MALI_FALSE;
+
+ /*
+ * The slot must be free already. If not, then something went
+ * wrong in the validate path.
+ */
+ OSK_ASSERT(!dq->queue[s]);
+
+ dq->queue[s] = katom;
+ beenthere("queued %p slot %d", (void *)katom, s);
+
+ return MALI_TRUE;
+}
+KBASE_EXPORT_TEST_API(jd_add_dep)
+
+/*
+ * This function only computes the address of the first possible
+ * atom. It doesn't mean it's actually valid (jd_validate_atom takes
+ * care of that).
+ */
+STATIC INLINE base_jd_atom *jd_get_first_atom(kbase_jd_context *ctx,
+ kbase_jd_bag *bag)
+{
+ /* Check that offset is within pool */
+ if ((bag->offset + sizeof(base_jd_atom)) > ctx->pool_size)
+ return NULL;
+
+ return (base_jd_atom *)((char *)ctx->pool + bag->offset);
+}
+KBASE_EXPORT_TEST_API(jd_get_first_atom)
+
+/*
+ * Same as with jd_get_first_atom, but for any subsequent atom.
+ */
+STATIC INLINE base_jd_atom *jd_get_next_atom(kbase_jd_atom *katom)
+{
+ return (katom->core_req & BASE_JD_REQ_EXTERNAL_RESOURCES) ? (base_jd_atom *)base_jd_get_external_resource(katom->user_atom, katom->nr_extres) :
+ (base_jd_atom *)base_jd_get_atom_syncset(katom->user_atom, katom->nr_syncsets);
+}
+KBASE_EXPORT_TEST_API(jd_get_next_atom)
+
+#ifdef CONFIG_KDS
+static void kds_dep_clear(void * callback_parameter, void * callback_extra_parameter)
+{
+ kbase_jd_atom * katom;
+ kbase_jd_context * ctx;
+
+ katom = (kbase_jd_atom*)callback_parameter;
+ OSK_ASSERT(katom);
+ ctx = &katom->kctx->jctx;
+
+ osk_mutex_lock(&ctx->lock);
+
+ OSK_ASSERT(katom->kds_dep_satisfied == MALI_FALSE);
+
+ /* This atom's KDS dependency has now been met */
+ katom->kds_dep_satisfied = MALI_TRUE;
+
+ /* Check whether the atom's other dependencies were already met */
+ if (ctx->dep_queue.queue[katom->pre_dep.dep[0]] != katom &&
+ ctx->dep_queue.queue[katom->pre_dep.dep[1]] != katom)
+ {
+ /* katom dep complete, add to JS */
+ mali_bool resched;
+
+ resched = kbasep_js_add_job( katom->kctx, katom );
+
+ if (resched)
+ {
+ kbasep_js_try_schedule_head_ctx(katom->kctx->kbdev);
+ }
+ }
+ osk_mutex_unlock(&ctx->lock);
+}
+#endif
+
+#ifdef CONFIG_DMA_SHARED_BUFFER
+static mali_error kbase_jd_umm_map(struct kbase_context * kctx, struct kbase_va_region * reg)
+{
+ struct sg_table * st;
+ struct scatterlist * s;
+ int i;
+ osk_phy_addr * pa;
+ mali_error err;
+
+ OSK_ASSERT(NULL == reg->imported_metadata.umm.st);
+ st = dma_buf_map_attachment(reg->imported_metadata.umm.dma_attachment, DMA_BIDIRECTIONAL);
+
+ if (!st)
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ /* save for later */
+ reg->imported_metadata.umm.st = st;
+
+ pa = kbase_get_phy_pages(reg);
+ OSK_ASSERT(pa);
+
+ for_each_sg(st->sgl, s, st->nents, i)
+ {
+ int j;
+ size_t pages = PFN_DOWN(sg_dma_len(s));
+
+ for (j = 0; j < pages; j++)
+ *pa++ = sg_dma_address(s) + (j << PAGE_SHIFT);
+ }
+
+ err = kbase_mmu_insert_pages(kctx, reg->start_pfn, kbase_get_phy_pages(reg), reg->nr_alloc_pages, reg->flags | KBASE_REG_GPU_WR | KBASE_REG_GPU_RD);
+
+ if (MALI_ERROR_NONE != err)
+ {
+ dma_buf_unmap_attachment(reg->imported_metadata.umm.dma_attachment, reg->imported_metadata.umm.st, DMA_BIDIRECTIONAL);
+ reg->imported_metadata.umm.st = NULL;
+ }
+
+ return err;
+}
+
+static void kbase_jd_umm_unmap(struct kbase_context * kctx, struct kbase_va_region * reg)
+{
+ OSK_ASSERT(kctx);
+ OSK_ASSERT(reg);
+ OSK_ASSERT(reg->imported_metadata.umm.dma_attachment);
+ OSK_ASSERT(reg->imported_metadata.umm.st);
+ kbase_mmu_teardown_pages(kctx, reg->start_pfn, reg->nr_alloc_pages);
+ dma_buf_unmap_attachment(reg->imported_metadata.umm.dma_attachment, reg->imported_metadata.umm.st, DMA_BIDIRECTIONAL);
+ reg->imported_metadata.umm.st = NULL;
+}
+#endif /* CONFIG_DMA_SHARED_BUFFER */
+
+void kbase_jd_post_external_resources(kbase_jd_atom * katom)
+{
+#ifdef CONFIG_DMA_SHARED_BUFFER
+ u32 res_no;
+#endif /* CONFIG_DMA_SHARED_BUFFER */
+
+ OSK_ASSERT(katom);
+ OSK_ASSERT(katom->core_req & BASE_JD_REQ_EXTERNAL_RESOURCES);
+
+#ifdef CONFIG_KDS
+ if (katom->kds_rset)
+ {
+ kds_resource_set_release(&katom->kds_rset);
+ }
+#endif
+
+#if defined(CONFIG_DMA_SHARED_BUFFER) || (MALI_DEBUG != 0)
+ /* Lock also used in debug mode just for lock order checking */
+ kbase_gpu_vm_lock(katom->kctx);
+#endif /* defined(CONFIG_DMA_SHARED_BUFFER) || (MALI_DEBUG != 0) */
+#ifdef CONFIG_DMA_SHARED_BUFFER
+ res_no = katom->nr_extres;
+ while (res_no-- > 0)
+ {
+ base_external_resource * res;
+ kbase_va_region * reg;
+
+ res = base_jd_get_external_resource(katom->user_atom, res_no);
+ reg = kbase_region_lookup(katom->kctx, res->ext_resource & ~BASE_EXT_RES_ACCESS_EXCLUSIVE);
+ /* if reg wasn't found then it has been freed while the job ran */
+ if (reg)
+ {
+ if (1 == reg->imported_metadata.umm.current_mapping_usage_count--)
+ {
+ /* last job using */
+ kbase_jd_umm_unmap(katom->kctx, reg);
+ }
+ }
+ }
+#endif /* CONFIG_DMA_SHARED_BUFFER */
+#if defined(CONFIG_DMA_SHARED_BUFFER) || (MALI_DEBUG != 0)
+ /* Lock also used in debug mode just for lock order checking */
+ kbase_gpu_vm_unlock(katom->kctx);
+#endif /* defined(CONFIG_DMA_SHARED_BUFFER) || (MALI_DEBUG != 0) */
+}
+
+static mali_error kbase_jd_pre_external_resources(kbase_jd_atom * katom)
+{
+ mali_error err_ret_val = MALI_ERROR_FUNCTION_FAILED;
+ u32 res_no;
+#ifdef CONFIG_KDS
+ u32 kds_res_count = 0;
+ struct kds_resource ** kds_resources = NULL;
+ unsigned long * kds_access_bitmap = NULL;
+#endif /* CONFIG_KDS */
+
+ OSK_ASSERT(katom);
+ OSK_ASSERT(katom->core_req & BASE_JD_REQ_EXTERNAL_RESOURCES);
+
+ if (!katom->nr_extres)
+ {
+ /* no resources encoded, early out */
+ return MALI_ERROR_NONE;
+ }
+
+#ifdef CONFIG_KDS
+ /* assume we have to wait for all */
+ kds_resources = osk_malloc(sizeof(struct kds_resource *) * katom->nr_extres);
+ if (NULL == kds_resources)
+ {
+ err_ret_val = MALI_ERROR_OUT_OF_MEMORY;
+ goto early_err_out;
+ }
+
+ kds_access_bitmap = osk_calloc(sizeof(unsigned long) * ((katom->nr_extres + OSK_BITS_PER_LONG - 1) / OSK_BITS_PER_LONG));
+ if (NULL == kds_access_bitmap)
+ {
+ err_ret_val = MALI_ERROR_OUT_OF_MEMORY;
+ goto early_err_out;
+ }
+#endif /* CONFIG_KDS */
+
+#if defined(CONFIG_DMA_SHARED_BUFFER) || (MALI_DEBUG != 0)
+ /* need to keep the GPU VM locked while we set up UMM buffers */
+ /* Lock also used in debug mode just for lock order checking */
+ kbase_gpu_vm_lock(katom->kctx);
+#endif /* defined(CONFIG_DMA_SHARED_BUFFER) || (MALI_DEBUG != 0) */
+
+ for (res_no = 0; res_no < katom->nr_extres; res_no++)
+ {
+ base_external_resource * res;
+ kbase_va_region * reg;
+
+ res = base_jd_get_external_resource(katom->user_atom, res_no);
+ reg = kbase_region_lookup(katom->kctx, res->ext_resource & ~BASE_EXT_RES_ACCESS_EXCLUSIVE);
+
+ /* did we find a matching region object? */
+ if (NULL == reg)
+ {
+ /* roll back */
+ goto failed_loop;
+ }
+
+ /* decide what needs to happen for this resource */
+ switch (reg->imported_type)
+ {
+ case BASE_TMEM_IMPORT_TYPE_UMP:
+ {
+#if defined(CONFIG_KDS) && (MALI_USE_UMP == 1)
+ struct kds_resource * kds_res;
+ kds_res = ump_dd_kds_resource_get(reg->imported_metadata.ump_handle);
+ if (kds_res)
+ {
+ kds_resources[kds_res_count] = kds_res;
+ if (res->ext_resource & BASE_EXT_RES_ACCESS_EXCLUSIVE)
+ osk_bitarray_set_bit(kds_res_count, kds_access_bitmap);
+ kds_res_count++;
+ }
+#endif /*defined(CONFIG_KDS) && (MALI_USE_UMP == 1)*/
+ break;
+ }
+#ifdef CONFIG_DMA_SHARED_BUFFER
+ case BASE_TMEM_IMPORT_TYPE_UMM:
+ {
+ reg->imported_metadata.umm.current_mapping_usage_count++;
+ if (1 == reg->imported_metadata.umm.current_mapping_usage_count)
+ {
+ err_ret_val = kbase_jd_umm_map(katom->kctx, reg);
+ if (MALI_ERROR_NONE != err_ret_val)
+ {
+ /* failed to map this buffer, roll back */
+ goto failed_loop;
+ }
+ }
+ break;
+ }
+#endif
+ default:
+ goto failed_loop;
+ }
+ }
+ /* successfully parsed the extres array */
+#if defined(CONFIG_DMA_SHARED_BUFFER) || (MALI_DEBUG != 0)
+ /* drop the vm lock before we call into kds */
+ /* Lock also used in debug mode just for lock order checking */
+ kbase_gpu_vm_unlock(katom->kctx);
+#endif /* defined(CONFIG_DMA_SHARED_BUFFER) || (MALI_DEBUG != 0) */
+
+#ifdef CONFIG_KDS
+ if (kds_res_count)
+ {
+ /* We have resources to wait for with kds */
+ katom->kds_dep_satisfied = MALI_FALSE;
+ if (kds_async_waitall(&katom->kds_rset, KDS_FLAG_LOCKED_IGNORE, &katom->kctx->jctx.kds_cb, katom, NULL, kds_res_count, kds_access_bitmap, kds_resources))
+ {
+ goto failed_kds_setup;
+ }
+ }
+ else
+ {
+ /* Nothing to wait for, so kds dep met */
+ katom->kds_dep_satisfied = MALI_TRUE;
+ }
+ osk_free(kds_resources);
+ osk_free(kds_access_bitmap);
+#endif /* CONFIG_KDS */
+
+ /* all done OK */
+ return MALI_ERROR_NONE;
+
+
+/* error handling section */
+
+#ifdef CONFIG_KDS
+failed_kds_setup:
+#endif /* CONFIG_KDS */
+#if defined(CONFIG_DMA_SHARED_BUFFER) || (MALI_DEBUG != 0)
+ /* lock before we unmap */
+ /* Lock also used in debug mode just for lock order checking */
+ kbase_gpu_vm_lock(katom->kctx);
+#endif /* defined(CONFIG_DMA_SHARED_BUFFER) || (MALI_DEBUG != 0) */
+
+failed_loop:
+#ifdef CONFIG_DMA_SHARED_BUFFER
+ /* undo the loop work */
+ while (res_no-- > 0)
+ {
+ base_external_resource * res;
+ kbase_va_region * reg;
+ res = base_jd_get_external_resource(katom->user_atom, res_no);
+ reg = kbase_region_lookup(katom->kctx, res->ext_resource & ~BASE_EXT_RES_ACCESS_EXCLUSIVE);
+ /* if reg wasn't found then it has been freed when we set up kds */
+ if (reg)
+ {
+ reg->imported_metadata.umm.current_mapping_usage_count--;
+ if (0 == reg->imported_metadata.umm.current_mapping_usage_count)
+ {
+ kbase_jd_umm_unmap(katom->kctx, reg);
+ }
+ }
+ }
+#endif /* CONFIG_DMA_SHARED_BUFFER */
+#if defined(CONFIG_DMA_SHARED_BUFFER) || (MALI_DEBUG != 0)
+ /* Lock also used in debug mode just for lock order checking */
+ kbase_gpu_vm_unlock(katom->kctx);
+#endif /* defined(CONFIG_DMA_SHARED_BUFFER) || (MALI_DEBUG != 0) */
+
+#ifdef CONFIG_KDS
+early_err_out:
+ if (kds_resources)
+ {
+ osk_free(kds_resources);
+ }
+ if (kds_access_bitmap)
+ {
+ osk_free(kds_access_bitmap);
+ }
+#endif /* CONFIG_KDS */
+ return err_ret_val;
+}
+/*
+ * This will check atom for correctness and if so, initialize its js policy.
+ */
+STATIC INLINE kbase_jd_atom *jd_validate_atom(struct kbase_context *kctx,
+ kbase_jd_bag *bag,
+ base_jd_atom *atom,
+ u32 *sem)
+{
+ kbase_jd_context *jctx = &kctx->jctx;
+ kbase_jd_atom *katom;
+ u32 nr_syncsets;
+ u32 nr_extres;
+ base_jd_core_req core_req;
+ base_jd_dep pre_dep;
+ int nice_priority;
+
+ /* Check the atom struct fits in the pool before we attempt to access it
+ Note: a bad bag->nr_atom could trigger this condition */
+ if(((char *)atom + sizeof(base_jd_atom)) > ((char *)jctx->pool + jctx->pool_size))
+ return NULL;
+
+ core_req = atom->core_req;
+ nr_syncsets = atom->nr_syncsets;
+ nr_extres = atom->nr_extres;
+ pre_dep = atom->pre_dep;
+
+ if (kbase_hw_has_issue(kctx->kbdev, BASE_HW_ISSUE_8987))
+ {
+ /* For this HW workaround, we scheduled differently on the 'ONLY_COMPUTE'
+ * flag, at the expense of ignoring the NSS flag.
+ *
+ * NOTE: We could allow the NSS flag still (and just ensure that we still
+ * submit on slot 2 when the NSS flag is set), but we don't because:
+ * - If we only have NSS contexts, the NSS jobs get all the cores, delaying
+ * a non-NSS context from getting cores for a long time.
+ * - A single compute context won't be subject to any timers anyway -
+ * only when there are >1 contexts (GLES *or* CL) will it get subject to
+ * timers.
+ */
+ core_req &= ~((base_jd_core_req)BASE_JD_REQ_NSS);
+ }
+
+ /*
+ * Check that dependencies are sensible: the atom cannot have
+ * pre-dependencies that are already in use by another atom.
+ */
+ if (jctx->dep_queue.queue[pre_dep.dep[0]] ||
+ jctx->dep_queue.queue[pre_dep.dep[1]])
+ return NULL;
+
+ /* Check for conflicting dependencies inside the bag */
+ if (dep_get_sem(sem, pre_dep.dep[0]) ||
+ dep_get_sem(sem, pre_dep.dep[1]))
+ return NULL;
+
+ dep_raise_sem(sem, pre_dep.dep[0]);
+ dep_raise_sem(sem, pre_dep.dep[1]);
+
+ /* Check that the whole atom fits within the pool. */
+ if (core_req & BASE_JD_REQ_EXTERNAL_RESOURCES)
+ {
+ /* extres integrity will be verified when we parse them */
+ if ((char*)base_jd_get_external_resource(atom, nr_extres) > ((char*)jctx->pool + jctx->pool_size))
+ {
+ return NULL;
+ }
+ }
+ else
+ {
+ /* syncsets integrity will be performed as we execute them */
+ if ((char *)base_jd_get_atom_syncset(atom, nr_syncsets) > ((char *)jctx->pool + jctx->pool_size))
+ return NULL;
+ }
+
+ /* We surely want to preallocate a pool of those, or have some
+ * kind of slab allocator around */
+ katom = osk_calloc(sizeof(*katom));
+ if (!katom)
+ return NULL; /* Ideally we should handle OOM more gracefully */
+
+ katom->user_atom = atom;
+ katom->pre_dep = pre_dep;
+ katom->post_dep = atom->post_dep;
+ katom->bag = bag;
+ katom->kctx = kctx;
+ katom->nr_syncsets = nr_syncsets;
+ katom->nr_extres = nr_extres;
+ katom->device_nr = atom->device_nr;
+ katom->affinity = 0;
+ katom->jc = atom->jc;
+ katom->coreref_state= KBASE_ATOM_COREREF_STATE_NO_CORES_REQUESTED;
+ katom->core_req = core_req;
+#ifdef CONFIG_KDS
+ /* Start by assuming that the KDS dependencies are satisfied,
+ * kbase_jd_pre_external_resources will correct this if there are dependencies */
+ katom->kds_dep_satisfied = MALI_TRUE;
+#endif
+
+ /*
+ * If the priority is increased we need to check the caller has security caps to do this, if
+ * prioirty is decreased then this is ok as the result will have no negative impact on other
+ * processes running.
+ */
+ katom->nice_prio = atom->prio;
+ if( 0 > katom->nice_prio)
+ {
+ mali_bool access_allowed;
+ access_allowed = kbase_security_has_capability(kctx, KBASE_SEC_MODIFY_PRIORITY, KBASE_SEC_FLAG_NOAUDIT);
+ if(!access_allowed)
+ {
+ /* For unprivileged processes - a negative priority is interpreted as zero */
+ katom->nice_prio = 0;
+ }
+ }
+
+ /* Scale priority range to use NICE range */
+ if(katom->nice_prio)
+ {
+ /* Remove sign for calculation */
+ nice_priority = katom->nice_prio+128;
+ /* Fixed point maths to scale from ..255 to 0..39 (NICE range with +20 offset) */
+ katom->nice_prio = (((20<<16)/128)*nice_priority)>>16;
+ }
+
+ /* pre-fill the event */
+ katom->event.event_code = BASE_JD_EVENT_DONE;
+ katom->event.data = katom;
+
+ if (katom->core_req & BASE_JD_REQ_EXTERNAL_RESOURCES)
+ {
+ /* handle what we need to do to access the external resources */
+ if (MALI_ERROR_NONE != kbase_jd_pre_external_resources(katom))
+ {
+ /* setup failed (no access, bad resource, unknown resource types, etc.) */
+ osk_free(katom);
+ return NULL;
+ }
+ }
+
+
+ /* Initialize the jobscheduler policy for this atom. Function will
+ * return error if the atom is malformed. Then immediately terminate
+ * the policy to free allocated resources and return error.
+ *
+ * Soft-jobs never enter the job scheduler so we don't initialise the policy for these
+ */
+ if ((katom->core_req & BASE_JD_REQ_SOFT_JOB) == 0)
+ {
+ kbasep_js_policy *js_policy = &(kctx->kbdev->js_data.policy);
+ if (MALI_ERROR_NONE != kbasep_js_policy_init_job( js_policy, kctx, katom ))
+ {
+ if ( katom->core_req & BASE_JD_REQ_EXTERNAL_RESOURCES )
+ {
+ kbase_jd_post_external_resources(katom);
+ }
+ osk_free( katom );
+ return NULL;
+ }
+ }
+
+ return katom;
+}
+KBASE_EXPORT_TEST_API(jd_validate_atom)
+
+static void kbase_jd_cancel_bag(kbase_context *kctx, kbase_jd_bag *bag,
+ base_jd_event_code code)
+{
+ bag->event.event_code = code;
+ kbase_event_post(kctx, &bag->event);
+}
+
+STATIC void kbase_jd_katom_dtor(kbase_event *event)
+{
+ kbase_jd_atom *katom = CONTAINER_OF(event, kbase_jd_atom, event);
+ kbase_context *kctx = katom->kctx;
+ kbasep_js_policy *js_policy = &(kctx->kbdev->js_data.policy);
+
+ /* Soft-jobs never enter the job scheduler (see jd_validate_atom) therefore we
+ * do not need to terminate any of these jobs in the scheduler. We could get a
+ * request here due to kbase_jd_validate_bag failing an atom in the bag when
+ * a soft-job has already been validated and added to the event list */
+ if ((katom->core_req & BASE_JD_REQ_SOFT_JOB) == 0)
+ {
+ kbasep_js_policy_term_job( js_policy, kctx, katom );
+ }
+
+ if ( katom->core_req & BASE_JD_REQ_EXTERNAL_RESOURCES )
+ {
+ kbase_jd_post_external_resources(katom);
+ }
+ osk_free(katom);
+}
+KBASE_EXPORT_TEST_API(kbase_jd_katom_dtor)
+
+STATIC mali_error kbase_jd_validate_bag(kbase_context *kctx,
+ kbase_jd_bag *bag,
+ osk_dlist *klistp)
+{
+ kbase_jd_context *jctx;
+ kbasep_js_kctx_info *js_kctx_info;
+ kbase_jd_atom *katom;
+ base_jd_atom *atom;
+ mali_error err = MALI_ERROR_NONE;
+ u32 sem[BASEP_JD_SEM_ARRAY_SIZE] = { 0 };
+ u32 i;
+
+ OSK_ASSERT( kctx != NULL );
+
+ /* Bags without any atoms are not allowed */
+ if (bag->nr_atoms == 0)
+ {
+ kbase_jd_cancel_bag(kctx, bag, BASE_JD_EVENT_BAG_INVALID);
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ jctx = &kctx->jctx;
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ atom = jd_get_first_atom(jctx, bag);
+ if (!atom)
+ {
+ /* Bad start... */
+ kbase_jd_cancel_bag(kctx, bag, BASE_JD_EVENT_BAG_INVALID);
+ err = MALI_ERROR_FUNCTION_FAILED;
+ goto out;
+ }
+
+ for (i = 0; i < bag->nr_atoms; i++)
+ {
+ katom = jd_validate_atom(kctx, bag, atom, sem);
+ if (!katom)
+ {
+ OSK_DLIST_EMPTY_LIST_REVERSE(klistp, kbase_event,
+ entry, kbase_jd_katom_dtor);
+ kbase_jd_cancel_bag(kctx, bag, BASE_JD_EVENT_BAG_INVALID);
+ err = MALI_ERROR_FUNCTION_FAILED;
+ goto out;
+ }
+
+ OSK_DLIST_PUSH_BACK(klistp, &katom->event,
+ kbase_event, entry);
+ atom = jd_get_next_atom(katom);
+ }
+
+out:
+ return err;
+}
+KBASE_EXPORT_TEST_API(kbase_jd_validate_bag)
+
+STATIC INLINE kbase_jd_atom *jd_resolve_dep(kbase_jd_atom *katom, u8 d, int zapping)
+{
+ u8 other_dep;
+ u8 dep;
+ kbase_jd_atom *dep_katom;
+ kbase_jd_context *ctx = &katom->kctx->jctx;
+
+ dep = katom->post_dep.dep[d];
+
+ if (!dep)
+ return NULL;
+
+ dep_clear_sem(ctx->dep_queue.sem, dep);
+
+ /* Get the atom that's waiting for us (if any), and remove it
+ * from this particular dependency queue */
+ dep_katom = ctx->dep_queue.queue[dep];
+
+ /* Case of a dangling dependency */
+ if (!dep_katom)
+ return NULL;
+
+ ctx->dep_queue.queue[dep] = NULL;
+
+ beenthere("removed %p from slot %d",
+ (void *)dep_katom, dep);
+
+#ifdef CONFIG_KDS
+ if (dep_katom->kds_dep_satisfied == MALI_FALSE)
+ {
+ /* The KDS dependency has not been satisfied yet */
+ return NULL;
+ }
+#endif
+
+ /* Find out if this atom is waiting for another job to be done.
+ * If it's not waiting anymore, put it on the run queue. */
+ if (dep_katom->pre_dep.dep[0] == dep)
+ other_dep = dep_katom->pre_dep.dep[1];
+ else
+ other_dep = dep_katom->pre_dep.dep[0];
+
+ /*
+ * The following line seem to confuse people, so here's the
+ * rational behind it:
+ *
+ * The queue hold pointers to atoms waiting for a single
+ * pre-dependency to be satisfied. Above, we've already
+ * satisfied a pre-dep for an atom (dep_katom). The next step
+ * is to check whether this atom is now free to run, or has to
+ * wait for another pre-dep to be satisfied.
+ *
+ * For a single entry, 3 possibilities:
+ *
+ * - It's a pointer to dep_katom -> the pre-dep has not been
+ * satisfied yet, and it cannot run immediately.
+ *
+ * - It's NULL -> the atom can be scheduled immediately, as
+ * the dependency has already been satisfied.
+ *
+ * - Neither of the above: this is the case of a dependency
+ * that has already been satisfied, and the slot reused by
+ * an incoming atom -> dep_katom can be run immediately.
+ */
+ if (ctx->dep_queue.queue[other_dep] != dep_katom)
+ return dep_katom;
+
+ /*
+ * We're on a killing spree. Cancel the additionnal
+ * dependency, and return the atom anyway. An unfortunate
+ * consequence is that userpace may receive notifications out
+ * of order WRT the dependency tree.
+ */
+ if (zapping)
+ {
+ ctx->dep_queue.queue[other_dep] = NULL;
+ return dep_katom;
+ }
+
+ beenthere("katom %p waiting for slot %d",
+ (void *)dep_katom, other_dep);
+ return NULL;
+}
+KBASE_EXPORT_TEST_API(jd_resolve_dep)
+
+/*
+ * Perform the necessary handling of an atom that has finished running
+ * on the GPU. The @a zapping parameter instruct the function to
+ * propagate the state of the completed atom to all the atoms that
+ * depend on it, directly or indirectly.
+ *
+ * This flag is used for error propagation in the "failed job", or
+ * when destroying a context.
+ *
+ * When not zapping, the caller must hold the kbasep_js_kctx_info::ctx::jsctx_mutex.
+ */
+STATIC mali_bool jd_done_nolock(kbase_jd_atom *katom, int zapping)
+{
+ kbase_jd_atom *dep_katom;
+ struct kbase_context *kctx = katom->kctx;
+ osk_dlist ts; /* traversal stack */
+ osk_dlist *tsp = &ts;
+ osk_dlist vl; /* visited list */
+ osk_dlist *vlp = &vl;
+ kbase_jd_atom *node;
+ base_jd_event_code event_code = katom->event.event_code;
+ mali_bool need_to_try_schedule_context = MALI_FALSE;
+
+ /*
+ * We're trying to achieve two goals here:
+ * - Eliminate dependency atoms very early so we can push real
+ * jobs to the HW
+ * - Avoid recursion which could result in a nice DoS from
+ * user-space.
+ *
+ * We use two lists here:
+ * - One as a stack (ts) to get rid of the recursion
+ * - The other to queue jobs that are either done or ready to
+ * run.
+ */
+ OSK_DLIST_INIT(tsp);
+ OSK_DLIST_INIT(vlp);
+
+ /* push */
+ OSK_DLIST_PUSH_BACK(tsp, &katom->event, kbase_event, entry);
+
+ while(!OSK_DLIST_IS_EMPTY(tsp))
+ {
+ /* pop */
+ node = OSK_DLIST_POP_BACK(tsp, kbase_jd_atom, event.entry);
+
+ if (node == katom ||
+ node->core_req == BASE_JD_REQ_DEP ||
+ zapping)
+ {
+ int i;
+ for (i = 0; i < 2; i++)
+ {
+ dep_katom = jd_resolve_dep(node, i, zapping);
+ if (dep_katom) /* push */
+ OSK_DLIST_PUSH_BACK(tsp,
+ &dep_katom->event,
+ kbase_event,
+ entry);
+ }
+ }
+
+ OSK_DLIST_PUSH_BACK(vlp, &node->event,
+ kbase_event, entry);
+ }
+
+ while(!OSK_DLIST_IS_EMPTY(vlp))
+ {
+ node = OSK_DLIST_POP_FRONT(vlp, kbase_jd_atom, event.entry);
+
+ if (node == katom ||
+ node->core_req == BASE_JD_REQ_DEP ||
+ (node->core_req & BASE_JD_REQ_SOFT_JOB) ||
+ zapping)
+ {
+ kbase_jd_bag *bag = node->bag;
+
+ /* If we're zapping stuff, propagate the event code */
+ if (zapping)
+ {
+ node->event.event_code = event_code;
+ }
+ else if (node->core_req & BASE_JD_REQ_SOFT_JOB)
+ {
+ kbase_process_soft_job( kctx, node );
+ }
+
+ /* This will signal our per-context worker
+ * thread that we're done with this katom. Any
+ * use of this katom after that point IS A
+ * ERROR!!! */
+ kbase_event_post(kctx, &node->event);
+ beenthere("done atom %p\n", (void*)node);
+
+ OSK_ASSERT( kctx->nr_outstanding_atoms > 0 );
+ if (--kctx->nr_outstanding_atoms < MAX_KCTX_OUTSTANDING_ATOMS)
+ {
+ osk_waitq_set(&kctx->complete_outstanding_waitq);
+ }
+ if (--bag->nr_atoms == 0)
+ {
+ if (bag->has_pm_ctx_reference)
+ {
+ /* This bag had a pm reference on the GPU, release it */
+ kbase_pm_context_idle(kctx->kbdev);
+ }
+ beenthere("done bag %p\n", (void*)bag);
+ /* This atom was the last, signal userspace */
+ kbase_event_post(kctx, &bag->event);
+ /* The bag may be freed by this point - it is a bug to try to access it after this point */
+ }
+
+ /* Decrement and check the TOTAL number of jobs. This includes
+ * those not tracked by the scheduler: 'not ready to run' and
+ * 'dependency-only' jobs. */
+ if (--kctx->jctx.job_nr == 0)
+ {
+ /* All events are safely queued now, and we can signal any waiter
+ * that we've got no more jobs (so we can be safely terminated) */
+ osk_waitq_set(&kctx->jctx.zero_jobs_waitq);
+ }
+ }
+ else
+ {
+ /* Queue an action about whether we should try scheduling a context */
+ need_to_try_schedule_context |= kbasep_js_add_job( kctx, node );
+ }
+ }
+
+ return need_to_try_schedule_context;
+}
+KBASE_EXPORT_TEST_API(jd_done_nolock)
+
+mali_error kbase_jd_submit(kbase_context *kctx, const kbase_uk_job_submit *user_bag)
+{
+ osk_dlist klist;
+ osk_dlist *klistp = &klist;
+ kbase_jd_context *jctx = &kctx->jctx;
+ kbase_jd_atom *katom;
+ kbase_jd_bag *bag;
+ mali_error err = MALI_ERROR_NONE;
+ int i = -1;
+ mali_bool need_to_try_schedule_context = MALI_FALSE;
+ kbase_device *kbdev;
+
+ /*
+ * kbase_jd_submit isn't expected to fail and so all errors with the jobs
+ * are reported by immediately falling them (through event system)
+ */
+ kbdev = kctx->kbdev;
+
+ beenthere("%s", "Enter");
+ bag = osk_malloc(sizeof(*bag));
+ if (NULL == bag)
+ {
+ err = MALI_ERROR_OUT_OF_MEMORY;
+ goto out_bag;
+ }
+
+ bag->core_restriction = user_bag->core_restriction;
+ bag->offset = user_bag->offset;
+ bag->nr_atoms = user_bag->nr_atoms;
+ bag->event.event_code = BASE_JD_EVENT_BAG_DONE;
+ bag->event.data = (void *)(uintptr_t)user_bag->bag_uaddr;
+ bag->has_pm_ctx_reference = MALI_FALSE;
+
+ osk_mutex_lock(&jctx->lock);
+ while (kctx->nr_outstanding_atoms >= MAX_KCTX_OUTSTANDING_ATOMS)
+ {
+ osk_mutex_unlock(&jctx->lock);
+ osk_waitq_wait(&kctx->complete_outstanding_waitq);
+ osk_mutex_lock(&jctx->lock);
+ }
+ /*
+ * Use a transient list to store all the validated atoms.
+ * Once we're sure nothing is wrong, there's no going back.
+ */
+ OSK_DLIST_INIT(klistp);
+
+ /* The above mutex lock provides necessary barrier to read this flag */
+ if ((kctx->jctx.sched_info.ctx.flags & KBASE_CTX_FLAG_SUBMIT_DISABLED) != 0)
+ {
+ OSK_PRINT_ERROR(OSK_BASE_JD, "Cancelled bag because context had SUBMIT_DISABLED set on it");
+ kbase_jd_cancel_bag(kctx, bag, BASE_JD_EVENT_BAG_INVALID);
+ err = MALI_ERROR_FUNCTION_FAILED;
+ goto out;
+ }
+
+ if (kbase_jd_validate_bag(kctx, bag, klistp))
+ {
+ err = MALI_ERROR_FUNCTION_FAILED;
+ goto out;
+ }
+
+ kctx->nr_outstanding_atoms += user_bag->nr_atoms;
+ if (kctx->nr_outstanding_atoms >= MAX_KCTX_OUTSTANDING_ATOMS )
+ {
+ osk_waitq_clear(&kctx->complete_outstanding_waitq);
+ }
+
+ while(!OSK_DLIST_IS_EMPTY(klistp))
+ {
+
+ katom = OSK_DLIST_POP_FRONT(klistp,
+ kbase_jd_atom, event.entry);
+ i++;
+
+ /* This is crucial. As jobs are processed in-order, we must
+ * indicate that any job with a pre-dep on this particular job
+ * must wait for its completion (indicated as a post-dep).
+ */
+ dep_raise_sem(jctx->dep_queue.sem, katom->post_dep.dep[0]);
+ dep_raise_sem(jctx->dep_queue.sem, katom->post_dep.dep[1]);
+
+ if (!(katom->core_req & BASE_JD_REQ_EXTERNAL_RESOURCES))
+ {
+ /* Process pre-exec syncsets before queueing */
+ kbase_pre_job_sync(kctx,
+ base_jd_get_atom_syncset(katom->user_atom, 0),
+ katom->nr_syncsets);
+ }
+
+ /* Update the TOTAL number of jobs. This includes those not tracked by
+ * the scheduler: 'not ready to run' and 'dependency-only' jobs. */
+ jctx->job_nr++;
+ /* Cause any future waiter-on-termination to wait until the jobs are
+ * finished */
+ osk_waitq_clear(&jctx->zero_jobs_waitq);
+ /* If no pre-dep has been set, then we're free to run
+ * the job immediately */
+ if ((jd_add_dep(jctx, katom, 0) | jd_add_dep(jctx, katom, 1)))
+ {
+ beenthere("queuing atom #%d(%p %p)", i,
+ (void *)katom, (void *)katom->user_atom);
+ continue;
+ }
+
+#ifdef CONFIG_KDS
+ if (!katom->kds_dep_satisfied)
+ {
+ /* Queue atom due to KDS dependency */
+ beenthere("queuing atom #%d(%p %p)", i,
+ (void *)katom, (void *)katom->user_atom);
+ continue;
+ }
+#endif
+
+ beenthere("running atom #%d(%p %p)", i,
+ (void *)katom, (void *)katom->user_atom);
+
+ if (katom->core_req & BASE_JD_REQ_SOFT_JOB)
+ {
+ kbase_process_soft_job( kctx, katom );
+ /* Pure software job, so resolve it immediately */
+ need_to_try_schedule_context |= jd_done_nolock(katom, 0);
+ }
+ else if (katom->core_req != BASE_JD_REQ_DEP)
+ {
+ need_to_try_schedule_context |= kbasep_js_add_job( kctx, katom );
+ }
+ else
+ {
+ /* This is a pure dependency. Resolve it immediately */
+ need_to_try_schedule_context |= jd_done_nolock(katom, 0);
+ }
+ }
+
+ /* This is an optimization: we only need to do this after processing all jobs
+ * resolved from this context. */
+ if ( need_to_try_schedule_context != MALI_FALSE )
+ {
+ kbasep_js_try_schedule_head_ctx( kbdev );
+ }
+out:
+ osk_mutex_unlock(&jctx->lock);
+out_bag:
+ beenthere("%s", "Exit");
+ return err;
+}
+KBASE_EXPORT_TEST_API(kbase_jd_submit)
+
+/**
+ * This function:
+ * - requeues the job from the runpool (if it was soft-stopped/removed from NEXT registers)
+ * - removes it from the system if it finished/failed/was cancelled.
+ * - resolves dependencies to add dependent jobs to the context, potentially starting them if necessary (which may add more references to the context)
+ * - releases the reference to the context from the no-longer-running job.
+ * - Handles retrying submission outside of IRQ context if it failed from within IRQ context.
+ */
+static void jd_done_worker(osk_workq_work *data)
+{
+ kbase_jd_atom *katom = CONTAINER_OF(data, kbase_jd_atom, work);
+ kbase_jd_context *jctx;
+ kbase_context *kctx;
+ kbasep_js_kctx_info *js_kctx_info;
+ kbasep_js_policy *js_policy;
+ kbase_device *kbdev;
+ kbasep_js_device_data *js_devdata;
+ int zapping;
+ u64 cache_jc = katom->jc;
+ base_jd_atom *cache_user_atom = katom->user_atom;
+
+ mali_bool retry_submit;
+ int retry_jobslot;
+
+ kctx = katom->kctx;
+ jctx = &kctx->jctx;
+ kbdev = kctx->kbdev;
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ js_devdata = &kbdev->js_data;
+ js_policy = &kbdev->js_data.policy;
+
+ KBASE_TRACE_ADD( kbdev, JD_DONE_WORKER, kctx, katom->user_atom, katom->jc, 0 );
+ /*
+ * Begin transaction on JD context and JS context
+ */
+ osk_mutex_lock( &jctx->lock );
+ osk_mutex_lock( &js_kctx_info->ctx.jsctx_mutex );
+
+ /* This worker only gets called on contexts that are scheduled *in*. This is
+ * because it only happens in response to an IRQ from a job that was
+ * running.
+ */
+ OSK_ASSERT( js_kctx_info->ctx.is_scheduled != MALI_FALSE );
+
+ /* Release cores this job was using (this might power down unused cores, and
+ * cause extra latency if a job submitted here - such as depenedent jobs -
+ * would use those cores) */
+ kbasep_js_job_check_deref_cores(kbdev, katom);
+
+ /* Grab the retry_submit state before the katom disappears */
+ retry_submit = kbasep_js_get_job_retry_submit_slot( katom, &retry_jobslot );
+
+ if (katom->event.event_code == BASE_JD_EVENT_STOPPED
+ || katom->event.event_code == BASE_JD_EVENT_REMOVED_FROM_NEXT )
+ {
+ /* Requeue the atom on soft-stop / removed from NEXT registers */
+ OSK_PRINT_INFO(OSK_BASE_JM, "JS: Soft Stopped/Removed from next %p on Ctx %p; Requeuing", kctx );
+
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+ kbasep_js_clear_job_retry_submit( katom );
+
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+ kbasep_js_policy_enqueue_job( js_policy, katom );
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+
+ /* A STOPPED/REMOVED job must cause a re-submit to happen, in case it
+ * was the last job left. Crucially, work items on work queues can run
+ * out of order e.g. on different CPUs, so being able to submit from
+ * the IRQ handler is not a good indication that we don't need to run
+ * jobs; the submitted job could be processed on the work-queue
+ * *before* the stopped job, even though it was submitted after. */
+ OSK_ASSERT( retry_submit != MALI_FALSE );
+
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+ }
+ else
+ {
+ /* Remove the job from the system for all other reasons */
+ mali_bool need_to_try_schedule_context;
+
+ kbasep_js_remove_job( kctx, katom );
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+ /* jd_done_nolock() requires the jsctx_mutex lock to be dropped */
+
+ zapping = (katom->event.event_code != BASE_JD_EVENT_DONE);
+ need_to_try_schedule_context = jd_done_nolock(katom, zapping);
+
+ /* This ctx is already scheduled in, so return value guarenteed FALSE */
+ OSK_ASSERT( need_to_try_schedule_context == MALI_FALSE );
+ }
+ /* katom may have been freed now, do not use! */
+
+ /*
+ * Transaction complete
+ */
+ osk_mutex_unlock( &jctx->lock );
+
+ /* Job is now no longer running, so can now safely release the context reference
+ * This potentially schedules out the context, schedules in a new one, and
+ * runs a new job on the new one */
+ kbasep_js_runpool_release_ctx( kbdev, kctx );
+
+ /* Submit on any slots that might've had atoms blocked by the affinity of
+ the completed atom. */
+ kbase_js_affinity_submit_to_blocked_slots( kbdev );
+
+ /* If the IRQ handler failed to get a job from the policy, try again from
+ * outside the IRQ handler */
+ if ( retry_submit != MALI_FALSE )
+ {
+ KBASE_TRACE_ADD_SLOT( kbdev, JD_DONE_TRY_RUN_NEXT_JOB, kctx, cache_user_atom, cache_jc, retry_jobslot );
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+ kbasep_js_try_run_next_job_on_slot( kbdev, retry_jobslot );
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+ }
+ KBASE_TRACE_ADD( kbdev, JD_DONE_WORKER_END, kctx, cache_user_atom, cache_jc, 0 );
+}
+
+/**
+ * Work queue job cancel function
+ * Only called as part of 'Zapping' a context (which occurs on termination)
+ * Operates serially with the jd_done_worker() on the work queue.
+ *
+ * This can only be called on contexts that aren't scheduled.
+ *
+ * @note We don't need to release most of the resources that would occur on
+ * kbase_jd_done() or jd_done_worker(), because the atoms here must not be
+ * running (by virtue of only being called on contexts that aren't
+ * scheduled). The only resources that are an exception to this are:
+ * - those held by kbasep_js_job_check_ref_cores(), because these resources are
+ * held for non-running atoms as well as running atoms.
+ */
+static void jd_cancel_worker(osk_workq_work *data)
+{
+ kbase_jd_atom *katom = CONTAINER_OF(data, kbase_jd_atom, work);
+ kbase_jd_context *jctx;
+ kbase_context *kctx;
+ kbasep_js_kctx_info *js_kctx_info;
+ mali_bool need_to_try_schedule_context;
+
+ kctx = katom->kctx;
+ jctx = &kctx->jctx;
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ {
+ kbase_device *kbdev = kctx->kbdev;
+ KBASE_TRACE_ADD( kbdev, JD_CANCEL_WORKER, kctx, katom->user_atom, katom->jc, 0 );
+ }
+
+ /* This only gets called on contexts that are scheduled out. Hence, we must
+ * make sure we don't de-ref the number of running jobs (there aren't
+ * any), nor must we try to schedule out the context (it's already
+ * scheduled out).
+ */
+ OSK_ASSERT( js_kctx_info->ctx.is_scheduled == MALI_FALSE );
+
+ /* Release cores this job was using (this might power down unused cores) */
+ kbasep_js_job_check_deref_cores(kctx->kbdev, katom);
+
+ /* Scheduler: Remove the job from the system */
+ osk_mutex_lock( &js_kctx_info->ctx.jsctx_mutex );
+ kbasep_js_remove_job( kctx, katom );
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+
+ osk_mutex_lock(&jctx->lock);
+
+ /* Always enable zapping */
+ need_to_try_schedule_context = jd_done_nolock(katom, 1);
+ /* Because we're zapping, we're not adding any more jobs to this ctx, so no need to
+ * schedule the context. There's also no need for the jsctx_mutex to have been taken
+ * around this too. */
+ OSK_ASSERT( need_to_try_schedule_context == MALI_FALSE );
+
+ /* katom may have been freed now, do not use! */
+ osk_mutex_unlock(&jctx->lock);
+
+}
+
+/**
+ * @brief Complete a job that has been removed from the Hardware
+ *
+ * This must be used whenever a job has been removed from the Hardware, e.g.:
+ * - An IRQ indicates that the job finished (for both error and 'done' codes)
+ * - The job was evicted from the JSn_HEAD_NEXT registers during a Soft/Hard stop.
+ *
+ * Some work is carried out immediately, and the rest is deferred onto a workqueue
+ *
+ * This can be called safely from atomic context.
+ *
+ */
+void kbase_jd_done(kbase_jd_atom *katom, int slot_nr, kbasep_js_tick *end_timestamp, mali_bool start_new_jobs)
+{
+ kbase_context *kctx;
+ kbase_device *kbdev;
+ OSK_ASSERT(katom);
+ kctx = katom->kctx;
+ OSK_ASSERT(kctx);
+ kbdev = kctx->kbdev;
+ OSK_ASSERT(kbdev);
+
+ KBASE_TRACE_ADD( kbdev, JD_DONE, kctx, katom->user_atom, katom->jc, 0 );
+
+ kbasep_js_job_done_slot_irq( katom, slot_nr, end_timestamp, start_new_jobs );
+
+ osk_workq_work_init(&katom->work, jd_done_worker);
+ osk_workq_submit(&kctx->jctx.job_done_wq, &katom->work);
+}
+KBASE_EXPORT_TEST_API(kbase_jd_done)
+
+
+void kbase_jd_cancel(kbase_jd_atom *katom)
+{
+ kbase_context *kctx;
+ kbasep_js_kctx_info *js_kctx_info;
+ kbase_device *kbdev;
+
+ kctx = katom->kctx;
+ js_kctx_info = &kctx->jctx.sched_info;
+ kbdev = kctx->kbdev;
+
+ KBASE_TRACE_ADD( kbdev, JD_CANCEL, kctx, katom->user_atom, katom->jc, 0 );
+
+ /* This should only be done from a context that is not scheduled */
+ OSK_ASSERT( js_kctx_info->ctx.is_scheduled == MALI_FALSE );
+
+ katom->event.event_code = BASE_JD_EVENT_JOB_CANCELLED;
+
+ osk_workq_work_init(&katom->work, jd_cancel_worker);
+ osk_workq_submit(&kctx->jctx.job_done_wq, &katom->work);
+}
+
+void kbase_jd_flush_workqueues(kbase_context *kctx)
+{
+ kbase_device *kbdev;
+ int i;
+
+ OSK_ASSERT( kctx );
+
+ kbdev = kctx->kbdev;
+ OSK_ASSERT( kbdev );
+
+ osk_workq_flush( &kctx->jctx.job_done_wq );
+
+ /* Flush all workqueues, for simplicity */
+ for (i = 0; i < kbdev->nr_hw_address_spaces; i++)
+ {
+ osk_workq_flush( &kbdev->as[i].pf_wq );
+ }
+}
+
+typedef struct zap_reset_data
+{
+ /* The stages are:
+ * 1. The timer has never been called
+ * 2. The zap has timed out, all slots are soft-stopped - the GPU reset will happen.
+ * The GPU has been reset when kbdev->reset_waitq is signalled
+ *
+ * (-1 - The timer has been cancelled)
+ */
+ int stage;
+ kbase_device *kbdev;
+ osk_timer *timer;
+ osk_spinlock lock;
+} zap_reset_data;
+
+static void zap_timeout_callback(void *data)
+{
+ zap_reset_data *reset_data = (zap_reset_data*)data;
+ kbase_device *kbdev = reset_data->kbdev;
+
+ osk_spinlock_lock(&reset_data->lock);
+
+ if (reset_data->stage == -1)
+ {
+ goto out;
+ }
+
+ if (kbase_prepare_to_reset_gpu(kbdev))
+ {
+ kbase_reset_gpu(kbdev);
+ }
+
+ reset_data->stage = 2;
+
+out:
+ osk_spinlock_unlock(&reset_data->lock);
+}
+
+void kbase_jd_zap_context(kbase_context *kctx)
+{
+ kbase_device *kbdev;
+ osk_timer zap_timeout;
+ osk_error ret;
+ zap_reset_data reset_data;
+
+ OSK_ASSERT(kctx);
+
+ kbdev = kctx->kbdev;
+
+ KBASE_TRACE_ADD( kbdev, JD_ZAP_CONTEXT, kctx, NULL, 0u, 0u );
+ kbase_job_zap_context(kctx);
+
+ ret = osk_timer_on_stack_init(&zap_timeout);
+ if (ret != OSK_ERR_NONE)
+ {
+ goto skip_timeout;
+ }
+
+ ret = osk_spinlock_init(&reset_data.lock, OSK_LOCK_ORDER_JD_ZAP_CONTEXT);
+ if (ret != OSK_ERR_NONE)
+ {
+ osk_timer_on_stack_term(&zap_timeout);
+ goto skip_timeout;
+ }
+
+ reset_data.kbdev = kbdev;
+ reset_data.timer = &zap_timeout;
+ reset_data.stage = 1;
+ osk_timer_callback_set(&zap_timeout, zap_timeout_callback, &reset_data);
+ ret = osk_timer_start(&zap_timeout, ZAP_TIMEOUT);
+
+ if (ret != OSK_ERR_NONE)
+ {
+ osk_spinlock_term(&reset_data.lock);
+ osk_timer_on_stack_term(&zap_timeout);
+ goto skip_timeout;
+ }
+
+ /* If we jump to here then the zap timeout will not be active,
+ * so if the GPU hangs the driver will also hang. This will only
+ * happen if the driver is very resource starved.
+ */
+skip_timeout:
+
+ /* Wait for all jobs to finish, and for the context to be not-scheduled
+ * (due to kbase_job_zap_context(), we also guarentee it's not in the JS
+ * policy queue either */
+ osk_waitq_wait(&kctx->jctx.zero_jobs_waitq);
+ osk_waitq_wait(&kctx->jctx.sched_info.ctx.not_scheduled_waitq);
+
+ if (ret == OSK_ERR_NONE)
+ {
+ osk_spinlock_lock(&reset_data.lock);
+ if (reset_data.stage == 1)
+ {
+ /* The timer hasn't run yet - so cancel it */
+ reset_data.stage = -1;
+ }
+ osk_spinlock_unlock(&reset_data.lock);
+
+ osk_timer_stop(&zap_timeout);
+
+ if (reset_data.stage == 2)
+ {
+ /* The reset has already started.
+ * Wait for the reset to complete
+ */
+ osk_waitq_wait(&kbdev->reset_waitq);
+ }
+ osk_timer_on_stack_term(&zap_timeout);
+ osk_spinlock_term(&reset_data.lock);
+ }
+
+ OSK_PRINT_INFO(OSK_BASE_JM, "Zap: Finished Context %p", kctx );
+
+ /* Ensure that the signallers of the waitqs have finished */
+ osk_mutex_lock(&kctx->jctx.lock);
+ osk_mutex_lock(&kctx->jctx.sched_info.ctx.jsctx_mutex);
+ osk_mutex_unlock(&kctx->jctx.sched_info.ctx.jsctx_mutex);
+ osk_mutex_unlock(&kctx->jctx.lock);
+}
+KBASE_EXPORT_TEST_API(kbase_jd_zap_context)
+
+mali_error kbase_jd_init(struct kbase_context *kctx)
+{
+ void *kaddr;
+ int i;
+ mali_error mali_err;
+ osk_error osk_err;
+#ifdef CONFIG_KDS
+ int err;
+#endif
+
+ OSK_ASSERT(kctx);
+ OSK_ASSERT(NULL == kctx->jctx.pool);
+
+ kaddr = osk_vmalloc(BASEP_JCTX_RB_NRPAGES * OSK_PAGE_SIZE);
+ if (!kaddr)
+ {
+ mali_err = MALI_ERROR_OUT_OF_MEMORY;
+ goto out;
+ }
+ osk_err = osk_workq_init(&kctx->jctx.job_done_wq, "mali_jd", 0);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ mali_err = MALI_ERROR_OUT_OF_MEMORY;
+ goto out1;
+ }
+
+ for (i = 0; i < KBASE_JD_DEP_QUEUE_SIZE; i++)
+ kctx->jctx.dep_queue.queue[i] = NULL;
+
+ for (i = 0; i < BASEP_JD_SEM_ARRAY_SIZE; i++)
+ kctx->jctx.dep_queue.sem[i] = 0;
+
+ osk_err = osk_mutex_init(&kctx->jctx.lock, OSK_LOCK_ORDER_JCTX);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ mali_err = MALI_ERROR_FUNCTION_FAILED;
+ goto out2;
+ }
+
+ osk_err = osk_waitq_init(&kctx->jctx.zero_jobs_waitq);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ mali_err = MALI_ERROR_FUNCTION_FAILED;
+ goto out3;
+ }
+
+ osk_err = osk_spinlock_irq_init(&kctx->jctx.tb_lock, OSK_LOCK_ORDER_TB);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ mali_err = MALI_ERROR_FUNCTION_FAILED;
+ goto out4;
+ }
+
+#ifdef CONFIG_KDS
+ err = kds_callback_init(&kctx->jctx.kds_cb, 0, kds_dep_clear);
+ if (0 != err)
+ {
+ mali_err = MALI_ERROR_FUNCTION_FAILED;
+ goto out5;
+ }
+#endif
+
+ osk_waitq_set(&kctx->jctx.zero_jobs_waitq);
+
+ kctx->jctx.pool = kaddr;
+ kctx->jctx.pool_size = BASEP_JCTX_RB_NRPAGES * OSK_PAGE_SIZE;
+ kctx->jctx.job_nr = 0;
+
+ return MALI_ERROR_NONE;
+
+#ifdef CONFIG_KDS
+out5:
+ osk_spinlock_irq_term(&kctx->jctx.tb_lock);
+#endif
+out4:
+ osk_waitq_term(&kctx->jctx.zero_jobs_waitq);
+out3:
+ osk_mutex_term(&kctx->jctx.lock);
+out2:
+ osk_workq_term(&kctx->jctx.job_done_wq);
+out1:
+ osk_vfree(kaddr);
+out:
+ return mali_err;
+}
+KBASE_EXPORT_TEST_API(kbase_jd_init)
+
+void kbase_jd_exit(struct kbase_context *kctx)
+{
+ OSK_ASSERT(kctx);
+ /* Assert if kbase_jd_init has not been called before this function
+ (kbase_jd_init initializes the pool) */
+ OSK_ASSERT(kctx->jctx.pool);
+
+#ifdef CONFIG_KDS
+ kds_callback_term(&kctx->jctx.kds_cb);
+#endif
+ osk_spinlock_irq_term(&kctx->jctx.tb_lock);
+ /* Work queue is emptied by this */
+ osk_workq_term(&kctx->jctx.job_done_wq);
+ osk_waitq_term(&kctx->jctx.zero_jobs_waitq);
+ osk_vfree(kctx->jctx.pool);
+ osk_mutex_term(&kctx->jctx.lock);
+}
+KBASE_EXPORT_TEST_API(kbase_jd_exit)
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_jm.c
+ * Base kernel job manager APIs
+ */
+
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_midg_regmap.h>
+#include <kbase/src/common/mali_kbase_gator.h>
+#include <kbase/src/common/mali_kbase_js_affinity.h>
+#include <kbase/src/common/mali_kbase_8401_workaround.h>
+#include <kbase/src/common/mali_kbase_hw.h>
+
+#include "mali_kbase_jm.h"
+
+#define beenthere(f, a...) OSK_PRINT_INFO(OSK_BASE_JM, "%s:" f, __func__, ##a)
+
+static void kbasep_try_reset_gpu_early(kbase_device *kbdev);
+
+static void kbase_job_hw_submit(kbase_device *kbdev, kbase_jd_atom *katom, int js)
+{
+ kbase_context *kctx;
+ u32 cfg;
+ u64 jc_head = katom->jc;
+
+ OSK_ASSERT(kbdev);
+ OSK_ASSERT(katom);
+
+ kctx = katom->kctx;
+
+ /* Command register must be available */
+ OSK_ASSERT(kbasep_jm_is_js_free(kbdev, js, kctx));
+ /* Affinity is not violating */
+ kbase_js_debug_log_current_affinities( kbdev );
+ OSK_ASSERT(!kbase_js_affinity_would_violate(kbdev, js, katom->affinity));
+
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_LO), jc_head & 0xFFFFFFFF, kctx);
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_HI), jc_head >> 32, kctx);
+
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_AFFINITY_NEXT_LO), katom->affinity & 0xFFFFFFFF, kctx);
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_AFFINITY_NEXT_HI), katom->affinity >> 32, kctx);
+
+ /* start MMU, medium priority, cache clean/flush on end, clean/flush on start */
+ cfg = kctx->as_nr | JSn_CONFIG_END_FLUSH_CLEAN_INVALIDATE | JSn_CONFIG_START_MMU
+ | JSn_CONFIG_START_FLUSH_CLEAN_INVALIDATE | JSn_CONFIG_THREAD_PRI(8);
+
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_CONFIG_NEXT), cfg, kctx);
+
+ /* Write an approximate start timestamp.
+ * It's approximate because there might be a job in the HEAD register. In
+ * such cases, we'll try to make a better approximation in the IRQ handler
+ * (up to the KBASE_JS_IRQ_THROTTLE_TIME_US). */
+ katom->start_timestamp = kbasep_js_get_js_ticks();
+
+ /* GO ! */
+ OSK_PRINT_INFO(OSK_BASE_JM, "JS: Submitting atom %p from ctx %p to js[%d] with head=0x%llx, affinity=0x%llx",
+ katom, kctx, js, jc_head, katom->affinity );
+
+ KBASE_TRACE_ADD_SLOT_INFO( kbdev, JM_SUBMIT, kctx, katom->user_atom, jc_head, js, (u32)katom->affinity );
+
+#if MALI_GATOR_SUPPORT
+ kbase_trace_mali_job_slots_event(GATOR_MAKE_EVENT(GATOR_JOB_SLOT_START, js));
+#endif
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_COMMAND_NEXT), JSn_COMMAND_START, katom->kctx);
+}
+
+void kbase_job_submit_nolock(kbase_device *kbdev, kbase_jd_atom *katom, int js)
+{
+ kbase_jm_slot *jm_slots;
+
+ OSK_ASSERT(kbdev);
+
+ jm_slots = kbdev->jm_slots;
+
+ /*
+ * We can have:
+ * - one job already done (pending interrupt),
+ * - one running,
+ * - one ready to be run.
+ * Hence a maximum of 3 inflight jobs. We have a 4 job
+ * queue, which I hope will be enough...
+ */
+ kbasep_jm_enqueue_submit_slot( &jm_slots[js], katom );
+ kbase_job_hw_submit(kbdev, katom, js);
+}
+
+void kbase_job_done_slot(kbase_device *kbdev, int s, u32 completion_code, u64 job_tail, kbasep_js_tick *end_timestamp)
+{
+ kbase_jm_slot *slot;
+ kbase_jd_atom *katom;
+ mali_addr64 jc_head;
+ kbase_context *kctx;
+
+ OSK_ASSERT(kbdev);
+
+ /* IMPORTANT: this function must only contain work necessary to complete a
+ * job from a Real IRQ (and not 'fake' completion, e.g. from
+ * Soft-stop). For general work that must happen no matter how the job was
+ * removed from the hardware, place it in kbase_jd_done() */
+
+ slot = &kbdev->jm_slots[s];
+ katom = kbasep_jm_dequeue_submit_slot( slot );
+
+ /* If the katom completed is because it's a dummy job for HW workarounds, then take no further action */
+ if(kbasep_jm_is_dummy_workaround_job(kbdev, katom))
+ {
+ KBASE_TRACE_ADD_SLOT_INFO( kbdev, JM_JOB_DONE, NULL, NULL, 0, s, completion_code );
+ return;
+ }
+
+ jc_head = katom->jc;
+ kctx = katom->kctx;
+
+ KBASE_TRACE_ADD_SLOT_INFO( kbdev, JM_JOB_DONE, kctx, katom->user_atom, jc_head, s, completion_code );
+
+ if ( completion_code != BASE_JD_EVENT_DONE && completion_code != BASE_JD_EVENT_STOPPED )
+ {
+
+#if KBASE_TRACE_DUMP_ON_JOB_SLOT_ERROR != 0
+ KBASE_TRACE_DUMP( kbdev );
+#endif
+ }
+ if (job_tail != 0)
+ {
+ mali_bool was_updated = (job_tail != jc_head);
+ /* Some of the job has been executed, so we update the job chain address to where we should resume from */
+ katom->jc = job_tail;
+ if ( was_updated )
+ {
+ KBASE_TRACE_ADD_SLOT( kbdev, JM_UPDATE_HEAD, kctx, katom->user_atom, job_tail, s );
+ }
+ }
+
+ /* Only update the event code for jobs that weren't cancelled */
+ if ( katom->event.event_code != BASE_JD_EVENT_JOB_CANCELLED )
+ {
+ katom->event.event_code = completion_code;
+ }
+ kbase_device_trace_register_access(kctx, REG_WRITE , JOB_CONTROL_REG(JOB_IRQ_CLEAR), 1 << s);
+
+ /* Complete the job, with start_new_jobs = MALI_TRUE
+ *
+ * Also defer remaining work onto the workqueue:
+ * - Re-queue Soft-stopped jobs
+ * - For any other jobs, queue the job back into the dependency system
+ * - Schedule out the parent context if necessary, and schedule a new one in.
+ */
+ kbase_jd_done( katom, s, end_timestamp, MALI_TRUE );
+}
+
+/**
+ * Update the start_timestamp of the job currently in the HEAD, based on the
+ * fact that we got an IRQ for the previous set of completed jobs.
+ *
+ * The estimate also takes into account the KBASE_JS_IRQ_THROTTLE_TIME_US and
+ * the time the job was submitted, to work out the best estimate (which might
+ * still result in an over-estimate to the calculated time spent)
+ */
+STATIC void kbasep_job_slot_update_head_start_timestamp( kbase_device *kbdev, kbase_jm_slot *slot, kbasep_js_tick end_timestamp )
+{
+ OSK_ASSERT(slot);
+
+ if ( kbasep_jm_nr_jobs_submitted( slot ) > 0 )
+ {
+ kbase_jd_atom *katom;
+ kbasep_js_tick new_timestamp;
+ katom = kbasep_jm_peek_idx_submit_slot( slot, 0 ); /* The atom in the HEAD */
+
+ OSK_ASSERT( katom != NULL );
+
+ if ( kbasep_jm_is_dummy_workaround_job( kbdev, katom ) != MALI_FALSE )
+ {
+ /* Don't access the members of HW workaround 'dummy' jobs */
+ return;
+ }
+
+ /* Account for any IRQ Throttle time - makes an overestimate of the time spent by the job */
+ new_timestamp = end_timestamp - kbasep_js_convert_js_us_to_ticks(KBASE_JS_IRQ_THROTTLE_TIME_US);
+ if ( kbasep_js_ticks_after( new_timestamp, katom->start_timestamp ) )
+ {
+ /* Only update the timestamp if it's a better estimate than what's currently stored.
+ * This is because our estimate that accounts for the throttle time may be too much
+ * of an overestimate */
+ katom->start_timestamp = new_timestamp;
+ }
+ }
+}
+
+void kbase_job_done(kbase_device *kbdev, u32 done)
+{
+ int i;
+ u32 count = 0;
+ kbasep_js_tick end_timestamp = kbasep_js_get_js_ticks();
+
+ OSK_ASSERT(kbdev);
+
+ KBASE_TRACE_ADD( kbdev, JM_IRQ, NULL, NULL, 0, done );
+
+ OSK_MEMSET( &kbdev->slot_submit_count_irq[0], 0, sizeof(kbdev->slot_submit_count_irq) );
+
+ /* write irq throttle register, this will prevent irqs from occurring until
+ * the given number of gpu clock cycles have passed */
+ {
+ u32 irq_throttle_cycles = osk_atomic_get( &kbdev->irq_throttle_cycles );
+ kbase_reg_write( kbdev, JOB_CONTROL_REG( JOB_IRQ_THROTTLE ), irq_throttle_cycles, NULL );
+ }
+
+ while (done) {
+ kbase_jm_slot *slot;
+ u32 failed = done >> 16;
+
+ /* treat failed slots as finished slots */
+ u32 finished = (done & 0xFFFF) | failed;
+
+ /* Note: This is inherently unfair, as we always check
+ * for lower numbered interrupts before the higher
+ * numbered ones.*/
+ i = osk_find_first_set_bit(finished);
+ OSK_ASSERT(i >= 0);
+
+ slot = kbase_job_slot_lock(kbdev, i);
+
+ do {
+ int nr_done;
+ u32 active;
+ u32 completion_code = BASE_JD_EVENT_DONE; /* assume OK */
+ u64 job_tail = 0;
+
+ if (failed & (1u << i))
+ {
+ /* read out the job slot status code if the job slot reported failure */
+ completion_code = kbase_reg_read(kbdev, JOB_SLOT_REG(i, JSn_STATUS), NULL);
+
+ switch(completion_code)
+ {
+ case BASE_JD_EVENT_STOPPED:
+#if MALI_GATOR_SUPPORT
+ kbase_trace_mali_job_slots_event(GATOR_MAKE_EVENT(GATOR_JOB_SLOT_SOFT_STOPPED, i));
+#endif
+ /* Soft-stopped job - read the value of JS<n>_TAIL so that the job chain can be resumed */
+ job_tail = (u64)kbase_reg_read(kbdev, JOB_SLOT_REG(i, JSn_TAIL_LO), NULL) |
+ ((u64)kbase_reg_read(kbdev, JOB_SLOT_REG(i, JSn_TAIL_HI), NULL) << 32);
+ break;
+ default:
+ OSK_PRINT_WARN(OSK_BASE_JD, "error detected from slot %d, job status 0x%08x (%s)",
+ i, completion_code, kbase_exception_name(completion_code));
+ }
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_6787))
+ {
+ /* cache flush when jobs complete with non-done codes */
+ /* use GPU_COMMAND completion solution */
+ /* clean & invalidate the caches */
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_COMMAND), 8, NULL);
+
+ /* wait for cache flush to complete before continuing */
+ while((kbase_reg_read(kbdev, GPU_CONTROL_REG(GPU_IRQ_RAWSTAT), NULL) & CLEAN_CACHES_COMPLETED) == 0);
+ /* clear the CLEAN_CACHES_COMPLETED irq*/
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_IRQ_CLEAR), CLEAN_CACHES_COMPLETED, NULL);
+ }
+ }
+
+ kbase_reg_write(kbdev, JOB_CONTROL_REG(JOB_IRQ_CLEAR), done & ((1 << i) | (1 << (i + 16))), NULL);
+ active = kbase_reg_read(kbdev, JOB_CONTROL_REG(JOB_IRQ_JS_STATE), NULL);
+
+ if (((active >> i) & 1) == 0 && (((done >> (i+16)) & 1) == 0))
+ {
+ /* There is a potential race we must work around:
+ *
+ * 1. A job slot has a job in both current and next registers
+ * 2. The job in current completes successfully, the IRQ handler reads RAWSTAT
+ * and calls this function with the relevant bit set in "done"
+ * 3. The job in the next registers becomes the current job on the GPU
+ * 4. Sometime before the JOB_IRQ_CLEAR line above the job on the GPU _fails_
+ * 5. The IRQ_CLEAR clears the done bit but not the failed bit. This atomically sets
+ * JOB_IRQ_JS_STATE. However since both jobs have now completed the relevant bits
+ * for the slot are set to 0.
+ *
+ * If we now did nothing then we'd incorrectly assume that _both_ jobs had completed
+ * successfully (since we haven't yet observed the fail bit being set in RAWSTAT).
+ *
+ * So at this point if there are no active jobs left we check to see if RAWSTAT has a failure
+ * bit set for the job slot. If it does we know that there has been a new failure that we
+ * didn't previously know about, so we make sure that we record this in active (but we wait
+ * for the next loop to deal with it).
+ *
+ * If we were handling a job failure (i.e. done has the relevant high bit set) then we know that
+ * the value read back from JOB_IRQ_JS_STATE is the correct number of remaining jobs because
+ * the failed job will have prevented any futher jobs from starting execution.
+ */
+ u32 rawstat = kbase_reg_read(kbdev, JOB_CONTROL_REG(JOB_IRQ_RAWSTAT), NULL);
+
+ if ((rawstat >> (i+16)) & 1)
+ {
+ /* There is a failed job that we've missed - add it back to active */
+ active |= (1u << i);
+ }
+ }
+
+ OSK_PRINT_INFO(OSK_BASE_JM, "Job ended with status 0x%08X\n", completion_code);
+
+ nr_done = kbasep_jm_nr_jobs_submitted( slot );
+ nr_done -= (active >> i) & 1;
+ nr_done -= (active >> (i + 16)) & 1;
+
+ if (nr_done <= 0)
+ {
+ OSK_PRINT_WARN(OSK_BASE_JM,
+ "Spurious interrupt on slot %d",
+ i);
+ goto spurious;
+ }
+
+ count += nr_done;
+
+ while (nr_done) {
+ if (nr_done == 1)
+ {
+ kbase_job_done_slot(kbdev, i, completion_code, job_tail, &end_timestamp);
+ }
+ else
+ {
+ /* More than one job has completed. Since this is not the last job being reported this time it
+ * must have passed. This is because the hardware will not allow further jobs in a job slot to
+ * complete until the faile job is cleared from the IRQ status.
+ */
+ kbase_job_done_slot(kbdev, i, BASE_JD_EVENT_DONE, 0, &end_timestamp);
+ }
+ nr_done--;
+ }
+
+spurious:
+ done = kbase_reg_read(kbdev, JOB_CONTROL_REG(JOB_IRQ_RAWSTAT), NULL);
+
+ failed = done >> 16;
+ finished = (done & 0xFFFF) | failed;
+ } while (finished & (1 << i));
+
+ kbasep_job_slot_update_head_start_timestamp( kbdev, slot, end_timestamp );
+
+ kbase_job_slot_unlock(kbdev, i);
+ }
+
+ if (osk_atomic_get(&kbdev->reset_gpu) == KBASE_RESET_GPU_COMMITTED)
+ {
+ /* If we're trying to reset the GPU then we might be able to do it early
+ * (without waiting for a timeout) because some jobs have completed
+ */
+ kbasep_try_reset_gpu_early(kbdev);
+ }
+
+ KBASE_TRACE_ADD( kbdev, JM_IRQ_END, NULL, NULL, 0, count );
+}
+KBASE_EXPORT_TEST_API(kbase_job_done)
+
+
+static mali_bool kbasep_soft_stop_allowed(kbase_device *kbdev, u16 core_reqs)
+{
+ mali_bool soft_stops_allowed = MALI_TRUE;
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8408))
+ {
+ if ((core_reqs & BASE_JD_REQ_T) != 0)
+ {
+ soft_stops_allowed = MALI_FALSE;
+ }
+ }
+ return soft_stops_allowed;
+}
+
+static mali_bool kbasep_hard_stop_allowed(kbase_device *kbdev, u16 core_reqs)
+{
+ mali_bool hard_stops_allowed = MALI_TRUE;
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8394))
+ {
+ if ((core_reqs & BASE_JD_REQ_T) != 0)
+ {
+ hard_stops_allowed = MALI_FALSE;
+ }
+ }
+ return hard_stops_allowed;
+}
+
+static void kbasep_job_slot_soft_or_hard_stop_do_action(kbase_device *kbdev, int js, u32 action,
+ u16 core_reqs, kbase_context *kctx )
+{
+#if KBASE_TRACE_ENABLE
+ u32 status_reg_before;
+ u64 job_in_head_before;
+ u32 status_reg_after;
+
+ /* Check the head pointer */
+ job_in_head_before = ((u64)kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_HEAD_LO), NULL))
+ | (((u64)kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_HEAD_HI), NULL)) << 32);
+ status_reg_before = kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_STATUS), NULL );
+#endif
+
+ if (action == JSn_COMMAND_SOFT_STOP)
+ {
+ mali_bool soft_stop_allowed = kbasep_soft_stop_allowed( kbdev, core_reqs );
+ if (!soft_stop_allowed)
+ {
+#if MALI_DEBUG != 0
+ OSK_PRINT(OSK_BASE_JM, "Attempt made to soft-stop a job that cannot be soft-stopped. core_reqs = 0x%X", (unsigned int) core_reqs);
+#endif
+ return;
+ }
+ }
+
+ if (action == JSn_COMMAND_HARD_STOP)
+ {
+ mali_bool hard_stop_allowed = kbasep_hard_stop_allowed( kbdev, core_reqs );
+ if (!hard_stop_allowed)
+ {
+ /* Jobs can be hard-stopped for the following reasons:
+ * * CFS decides the job has been running too long (and soft-stop has not occurred).
+ * In this case the GPU will be reset by CFS if the job remains on the GPU.
+ *
+ * * The context is destroyed, kbase_jd_zap_context will attempt to hard-stop the job. However
+ * it also has a watchdog which will cause the GPU to be reset if the job remains on the GPU.
+ *
+ * * An (unhandled) MMU fault occurred. As long as BASE_HW_ISSUE_8245 is defined then
+ * the GPU will be reset.
+ *
+ * All three cases result in the GPU being reset if the hard-stop fails,
+ * so it is safe to just return and ignore the hard-stop request.
+ */
+ OSK_PRINT_WARN(OSK_BASE_JM, "Attempt made to hard-stop a job that cannot be hard-stopped. core_reqs = 0x%X", (unsigned int) core_reqs);
+ return;
+ }
+ }
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8316) && action == JSn_COMMAND_SOFT_STOP)
+ {
+ int i;
+ kbase_jm_slot *slot;
+ slot = &kbdev->jm_slots[js];
+
+ for (i = 0; i < kbasep_jm_nr_jobs_submitted(slot); i++)
+ {
+ kbase_jd_atom *katom;
+ kbase_as * as;
+
+ katom = kbasep_jm_peek_idx_submit_slot(slot, i);
+
+ OSK_ASSERT(katom);
+
+ if ( kbasep_jm_is_dummy_workaround_job( kbdev, katom ) != MALI_FALSE )
+ {
+ /* Don't access the members of HW workaround 'dummy' jobs
+ *
+ * This assumes that such jobs can't cause HW_ISSUE_8316, and could only be blocked
+ * by other jobs causing HW_ISSUE_8316 (which will get poked/or eventually get killed) */
+ continue;
+ }
+
+ if ( !katom->poking )
+ {
+ OSK_ASSERT(katom->kctx);
+ OSK_ASSERT(katom->kctx->as_nr != KBASEP_AS_NR_INVALID);
+
+ katom->poking = 1;
+ as = &kbdev->as[katom->kctx->as_nr];
+ kbase_as_poking_timer_retain(as);
+ }
+ }
+ }
+
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_COMMAND), action, kctx);
+
+#if KBASE_TRACE_ENABLE
+ status_reg_after = kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_STATUS), NULL );
+ if (status_reg_after == BASE_JD_EVENT_ACTIVE)
+ {
+ kbase_jm_slot *slot;
+ kbase_jd_atom *head;
+ kbase_context *head_kctx;
+
+ slot = &kbdev->jm_slots[js];
+ head = kbasep_jm_peek_idx_submit_slot( slot, slot->submitted_nr-1 );
+ head_kctx = head->kctx;
+
+ /* We don't need to check kbasep_jm_is_dummy_workaround_job( head ) here:
+ * - Members are not indirected through
+ * - The members will all be zero anyway
+ */
+ if ( status_reg_before == BASE_JD_EVENT_ACTIVE )
+ {
+ KBASE_TRACE_ADD_SLOT( kbdev, JM_CHECK_HEAD, head_kctx, head->user_atom, job_in_head_before, js );
+ }
+ else
+ {
+ KBASE_TRACE_ADD_SLOT( kbdev, JM_CHECK_HEAD, NULL, NULL, 0, js );
+ }
+ if (action == JSn_COMMAND_SOFT_STOP)
+ {
+ KBASE_TRACE_ADD_SLOT( kbdev, JM_SOFTSTOP, head_kctx, head->user_atom, head->jc, js );
+ }
+ else
+ {
+ KBASE_TRACE_ADD_SLOT( kbdev, JM_HARDSTOP, head_kctx, head->user_atom, head->jc, js );
+ }
+ }
+ else
+ {
+ if ( status_reg_before == BASE_JD_EVENT_ACTIVE )
+ {
+ KBASE_TRACE_ADD_SLOT( kbdev, JM_CHECK_HEAD, NULL, NULL, job_in_head_before, js );
+ }
+ else
+ {
+ KBASE_TRACE_ADD_SLOT( kbdev, JM_CHECK_HEAD, NULL, NULL, 0, js );
+ }
+
+ if (action == JSn_COMMAND_SOFT_STOP)
+ {
+ KBASE_TRACE_ADD_SLOT( kbdev, JM_SOFTSTOP, NULL, NULL, 0, js );
+ }
+ else
+ {
+ KBASE_TRACE_ADD_SLOT( kbdev, JM_HARDSTOP, NULL, NULL, 0, js );
+ }
+ }
+#endif
+}
+
+/* Helper macros used by kbasep_job_slot_soft_or_hard_stop */
+#define JM_SLOT_MAX_JOB_SUBMIT_REGS 2
+#define JM_JOB_IS_CURRENT_JOB_INDEX(n) (1 == n) /* Index of the last job to process */
+#define JM_JOB_IS_NEXT_JOB_INDEX(n) (2 == n) /* Index of the prior to last job to process */
+
+/** Soft or hard-stop a slot
+ *
+ * This function safely ensures that the correct job is either hard or soft-stopped.
+ * It deals with evicting jobs from the next registers where appropriate.
+ *
+ * This does not attempt to stop or evict jobs that are 'dummy' jobs for HW workarounds.
+ *
+ * @param kbdex The kbase device
+ * @param kctx The context to soft/hard-stop job(s) from (or NULL is all jobs should be targeted)
+ * @param js The slot that the job(s) are on
+ * @param target_katom The atom that should be targeted (or NULL if all jobs from the context should be targeted)
+ * @param action The action to perform, either JSn_COMMAND_HARD_STOP or JSn_COMMAND_SOFT_STOP
+ */
+static void kbasep_job_slot_soft_or_hard_stop(kbase_device *kbdev, kbase_context *kctx, int js,
+ kbase_jd_atom *target_katom, u32 action)
+{
+ kbase_jd_atom *katom;
+ u8 i;
+ u8 jobs_submitted;
+ kbase_jm_slot *slot;
+ u16 core_reqs;
+
+
+ OSK_ASSERT(action == JSn_COMMAND_HARD_STOP || action == JSn_COMMAND_SOFT_STOP);
+ OSK_ASSERT(kbdev);
+
+ slot = &kbdev->jm_slots[js];
+ OSK_ASSERT(slot);
+
+ jobs_submitted = kbasep_jm_nr_jobs_submitted( slot );
+ KBASE_TRACE_ADD_SLOT_INFO( kbdev, JM_SLOT_SOFT_OR_HARD_STOP, kctx, NULL, 0u, js, jobs_submitted );
+
+ if (jobs_submitted > JM_SLOT_MAX_JOB_SUBMIT_REGS)
+ {
+ i = jobs_submitted - JM_SLOT_MAX_JOB_SUBMIT_REGS;
+ }
+ else
+ {
+ i = 0;
+ }
+
+ /* Loop through all jobs that have been submitted to the slot and haven't completed */
+ for(;i < jobs_submitted;i++)
+ {
+ katom = kbasep_jm_peek_idx_submit_slot( slot, i );
+
+ if (kctx && katom->kctx != kctx)
+ {
+ continue;
+ }
+ if (target_katom && katom != target_katom)
+ {
+ continue;
+ }
+ if ( kbasep_jm_is_dummy_workaround_job( kbdev, katom ) )
+ {
+ continue;
+ }
+
+ core_reqs = katom->core_req;
+
+ if (JM_JOB_IS_CURRENT_JOB_INDEX(jobs_submitted - i))
+ {
+ /* The last job in the slot, check if there is a job in the next register */
+ if (kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_COMMAND_NEXT), NULL) == 0)
+ {
+ kbasep_job_slot_soft_or_hard_stop_do_action(kbdev, js, action, core_reqs, katom->kctx);
+ }
+ else
+ {
+ /* The job is in the next registers */
+ beenthere("clearing job from next registers on slot %d", js);
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_COMMAND_NEXT), JSn_COMMAND_NOP, NULL);
+
+ /* Check to see if we did remove a job from the next registers */
+ if (kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_LO), NULL) != 0 ||
+ kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_HI), NULL) != 0)
+ {
+ /* The job was successfully cleared from the next registers, requeue it */
+ kbase_jd_atom *dequeued_katom = kbasep_jm_dequeue_tail_submit_slot( slot );
+ OSK_ASSERT(dequeued_katom == katom);
+ jobs_submitted --;
+
+ /* Set the next registers to NULL */
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_LO), 0, NULL);
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_HI), 0, NULL);
+
+ KBASE_TRACE_ADD_SLOT( kbdev, JM_SLOT_EVICT, dequeued_katom->kctx, dequeued_katom->user_atom, dequeued_katom->jc, js );
+
+ dequeued_katom->event.event_code = BASE_JD_EVENT_REMOVED_FROM_NEXT;
+ /* Complete the job, indicate it took no time, but require start_new_jobs == MALI_FALSE
+ * to prevent this slot being resubmitted to until we've dropped the lock */
+ kbase_jd_done(dequeued_katom, js, NULL, MALI_FALSE);
+ }
+ else
+ {
+ /* The job transitioned into the current registers before we managed to evict it,
+ * in this case we fall back to soft/hard-stopping the job */
+ beenthere("missed job in next register, soft/hard-stopping slot %d", js);
+ kbasep_job_slot_soft_or_hard_stop_do_action(kbdev, js, action, core_reqs, katom->kctx);
+ }
+ }
+ }
+ else if (JM_JOB_IS_NEXT_JOB_INDEX(jobs_submitted-i))
+ {
+ /* There's a job after this one, check to see if that job is in the next registers */
+ if (kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_COMMAND_NEXT), NULL) != 0)
+ {
+ kbase_jd_atom *check_next_atom;
+ /* It is - we should remove that job and soft/hard-stop the slot */
+
+ /* Only proceed when the next jobs isn't a HW workaround 'dummy' job
+ *
+ * This can't be an ASSERT due to MMU fault code:
+ * - This first hard-stops the job that caused the fault
+ * - Under HW Issue 8401, this inserts a dummy workaround job into NEXT
+ * - Under HW Issue 8245, it will then reset the GPU
+ * - This causes a Soft-stop to occur on all slots
+ * - By the time of the soft-stop, we may (depending on timing) still have:
+ * - The original job in HEAD, if it's not finished the hard-stop
+ * - The dummy workaround job in NEXT
+ *
+ * Other cases could be coded in future that cause back-to-back Soft/Hard
+ * stops with dummy workaround jobs in place, e.g. MMU handler code and Job
+ * Scheduler watchdog timer running in parallel.
+ *
+ * Note, the index i+1 is valid to peek from: i == jobs_submitted-2, therefore
+ * i+1 == jobs_submitted-1 */
+ check_next_atom = kbasep_jm_peek_idx_submit_slot( slot, i+1 );
+ if ( kbasep_jm_is_dummy_workaround_job( kbdev, check_next_atom ) != MALI_FALSE )
+ {
+ continue;
+ }
+
+ beenthere("clearing job from next registers on slot %d", js);
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_COMMAND_NEXT), JSn_COMMAND_NOP, NULL);
+
+ /* Check to see if we did remove a job from the next registers */
+ if (kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_LO), NULL) != 0 ||
+ kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_HI), NULL) != 0)
+ {
+ /* We did remove a job from the next registers, requeue it */
+ kbase_jd_atom *dequeued_katom = kbasep_jm_dequeue_tail_submit_slot( slot );
+ OSK_ASSERT(dequeued_katom != NULL);
+ jobs_submitted --;
+
+ /* Set the next registers to NULL */
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_LO), 0, NULL);
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_HI), 0, NULL);
+
+ KBASE_TRACE_ADD_SLOT( kbdev, JM_SLOT_EVICT, dequeued_katom->kctx, dequeued_katom->user_atom, dequeued_katom->jc, js );
+
+ dequeued_katom->event.event_code = BASE_JD_EVENT_REMOVED_FROM_NEXT;
+ /* Complete the job, indicate it took no time, but require start_new_jobs == MALI_FALSE
+ * to prevent this slot being resubmitted to until we've dropped the lock */
+ kbase_jd_done(dequeued_katom, js, NULL, MALI_FALSE);
+ }
+ else
+ {
+ /* We missed the job, that means the job we're interested in left the hardware before
+ * we managed to do anything, so we can proceed to the next job */
+ continue;
+ }
+
+ /* Next is now free, so we can soft/hard-stop the slot */
+ beenthere("soft/hard-stopped slot %d (there was a job in next which was successfully cleared)\n", js);
+ kbasep_job_slot_soft_or_hard_stop_do_action(kbdev, js, action, core_reqs, katom->kctx);
+ }
+ /* If there was no job in the next registers, then the job we were
+ * interested in has finished, so we need not take any action
+ */
+ }
+ }
+}
+
+void kbase_job_kill_jobs_from_context(kbase_context *kctx)
+{
+ kbase_device *kbdev;
+ int i;
+
+ OSK_ASSERT( kctx != NULL );
+ kbdev = kctx->kbdev;
+ OSK_ASSERT( kbdev != NULL );
+
+ /* Cancel any remaining running jobs for this kctx */
+ for (i = 0; i < kbdev->gpu_props.num_job_slots; i++)
+ {
+ kbase_job_slot_lock(kbdev, i);
+ kbase_job_slot_hardstop(kctx, i, NULL);
+ kbase_job_slot_unlock(kbdev, i);
+ }
+}
+
+void kbase_job_zap_context(kbase_context *kctx)
+{
+ kbase_device *kbdev;
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_kctx_info *js_kctx_info;
+ int i;
+ mali_bool evict_success;
+
+ OSK_ASSERT( kctx != NULL );
+ kbdev = kctx->kbdev;
+ OSK_ASSERT( kbdev != NULL );
+ js_devdata = &kbdev->js_data;
+ js_kctx_info = &kctx->jctx.sched_info;
+
+
+ /*
+ * Critical assumption: No more submission is possible outside of the
+ * workqueue. This is because the OS *must* prevent U/K calls (IOCTLs)
+ * whilst the kbase_context is terminating.
+ */
+
+
+ /* First, atomically do the following:
+ * - mark the context as dying
+ * - try to evict it from the policy queue */
+
+ osk_mutex_lock( &js_kctx_info->ctx.jsctx_mutex );
+ js_kctx_info->ctx.is_dying = MALI_TRUE;
+
+ OSK_PRINT_INFO(OSK_BASE_JM, "Zap: Try Evict Ctx %p", kctx );
+ osk_mutex_lock( &js_devdata->queue_mutex );
+ evict_success = kbasep_js_policy_try_evict_ctx( &js_devdata->policy, kctx );
+ osk_mutex_unlock( &js_devdata->queue_mutex );
+
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+
+ /* locks must be dropped by this point, to prevent deadlock on flush */
+ OSK_PRINT_INFO(OSK_BASE_JM, "Zap: Flush Workqueue Ctx %p", kctx );
+ KBASE_TRACE_ADD( kbdev, JM_FLUSH_WORKQS, kctx, NULL, 0u, 0u );
+ kbase_jd_flush_workqueues( kctx );
+ KBASE_TRACE_ADD( kbdev, JM_FLUSH_WORKQS_DONE, kctx, NULL, 0u, 0u );
+
+ /*
+ * At this point we know that:
+ * - If eviction succeeded, it was in the policy queue, but now no longer is
+ * - If eviction failed, then it wasn't in the policy queue. It is one of the following:
+ * - a. it didn't have any jobs, and so is not in the Policy Queue or the
+ * Run Pool (no work required)
+ * - b. it was in the process of a scheduling transaction - but this can only
+ * happen as a result of the work-queue. Two options:
+ * - i. it is now scheduled by the time of the flush - case d.
+ * - ii. it is evicted from the Run Pool due to having to roll-back a transaction
+ * - c. it is about to be scheduled out.
+ * - In this case, we've marked it as dying, so the schedule-out code
+ * marks all jobs for killing, evicts it from the Run Pool, and does *not*
+ * place it back on the Policy Queue. The workqueue flush ensures this has
+ * completed
+ * - d. it is scheduled, and may or may not be running jobs
+ * - e. it was scheduled, but didn't get scheduled out during flushing of
+ * the workqueues. By the time we obtain the jsctx_mutex again, it may've
+ * been scheduled out
+ *
+ *
+ * Also Note: No-one can now clear the not_scheduled_waitq, because the
+ * context is guarenteed to not be in the policy queue, and can never
+ * return to it either (because is_dying is set). The waitq may already by
+ * clear (due to it being scheduled), but the code below ensures that it
+ * will eventually get set (be descheduled).
+ */
+
+ osk_mutex_lock( &js_kctx_info->ctx.jsctx_mutex );
+ if ( evict_success != MALI_FALSE || js_kctx_info->ctx.is_scheduled == MALI_FALSE )
+ {
+ /* The following events require us to kill off remaining jobs and
+ * update PM book-keeping:
+ * - we evicted it correctly (it must have jobs to be in the Policy Queue)
+ *
+ * These events need no action:
+ * - Case a: it didn't have any jobs, and was never in the Queue
+ * - Case b-ii: scheduling transaction was partially rolled-back (this
+ * already cancels the jobs and pm-idles the ctx)
+ * - Case c: scheduled out and killing of all jobs completed on the work-queue (it's not in the Run Pool)
+ * - Case e: it was scheduled out after the workqueue was flushed, but
+ * before we re-obtained the jsctx_mutex. The jobs have already been
+ * cancelled (but the cancel may not have completed yet) and the PM has
+ * already been idled
+ */
+
+ KBASE_TRACE_ADD( kbdev, JM_ZAP_NON_SCHEDULED, kctx, NULL, 0u, js_kctx_info->ctx.is_scheduled );
+
+ OSK_PRINT_INFO(OSK_BASE_JM, "Zap: Ctx %p evict_success=%d, scheduled=%d", kctx, evict_success, js_kctx_info->ctx.is_scheduled );
+
+ if ( evict_success != MALI_FALSE )
+ {
+ /* Only cancel jobs and pm-idle when we evicted from the policy queue.
+ *
+ * Having is_dying set ensures that this kills, and doesn't requeue
+ *
+ * In addition, is_dying set ensure that this calls kbase_pm_context_idle().
+ * This is safe because the context is guaranteed to not be in the
+ * runpool, by virtue of it being evicted from the policy queue */
+ kbasep_js_runpool_requeue_or_kill_ctx( kbdev, kctx );
+ }
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+ }
+ else
+ {
+ mali_bool was_retained;
+ /* Didn't evict, but it is scheduled - it's in the Run Pool:
+ * Cases d and b(i) */
+ KBASE_TRACE_ADD( kbdev, JM_ZAP_SCHEDULED, kctx, NULL, 0u, js_kctx_info->ctx.is_scheduled );
+ OSK_PRINT_INFO(OSK_BASE_JM, "Zap: Ctx %p is in RunPool", kctx );
+
+ /* Disable the ctx from submitting any more jobs */
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+ kbasep_js_clear_submit_allowed( js_devdata, kctx );
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+
+ /* Retain and release the context whilst it is is now disallowed from submitting
+ * jobs - ensures that someone somewhere will be removing the context later on */
+ was_retained = kbasep_js_runpool_retain_ctx( kbdev, kctx );
+
+ /* Since it's scheduled and we have the jsctx_mutex, it must be retained successfully */
+ OSK_ASSERT( was_retained != MALI_FALSE );
+
+ OSK_PRINT_INFO(OSK_BASE_JM, "Zap: Ctx %p Kill Any Running jobs", kctx );
+ /* Cancel any remaining running jobs for this kctx - if any. Submit is disallowed
+ * which takes effect from the dropping of the runpool_irq lock above, so no more new
+ * jobs will appear after we do this. */
+ for (i = 0; i < kbdev->gpu_props.num_job_slots; i++)
+ {
+ kbase_job_slot_lock(kbdev, i);
+ kbase_job_slot_hardstop(kctx, i, NULL);
+ kbase_job_slot_unlock(kbdev, i);
+ }
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+
+ OSK_PRINT_INFO(OSK_BASE_JM, "Zap: Ctx %p Release (may or may not schedule out immediately)", kctx );
+ kbasep_js_runpool_release_ctx( kbdev, kctx );
+ }
+ KBASE_TRACE_ADD( kbdev, JM_ZAP_DONE, kctx, NULL, 0u, 0u );
+
+ /* After this, you must wait on both the kbase_jd_context::zero_jobs_waitq
+ * and the kbasep_js_kctx_info::ctx::is_scheduled_waitq - to wait for the
+ * jobs to be destroyed, and the context to be de-scheduled (if it was on
+ * the runpool).
+ *
+ * kbase_jd_zap_context() will do this. */
+}
+KBASE_EXPORT_TEST_API(kbase_job_zap_context)
+
+mali_error kbase_job_slot_init(kbase_device *kbdev)
+{
+ int i;
+ osk_error osk_err;
+
+ OSK_ASSERT(kbdev);
+
+ for (i = 0; i < kbdev->gpu_props.num_job_slots; i++)
+ {
+ osk_err = osk_spinlock_irq_init(&kbdev->jm_slots[i].lock, OSK_LOCK_ORDER_JSLOT);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ int j;
+ for (j = 0; j < i; j++)
+ {
+ osk_spinlock_irq_term(&kbdev->jm_slots[j].lock);
+ }
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+ kbasep_jm_init_submit_slot( &kbdev->jm_slots[i] );
+ }
+ return MALI_ERROR_NONE;
+}
+KBASE_EXPORT_TEST_API(kbase_job_slot_init)
+
+void kbase_job_slot_halt(kbase_device *kbdev)
+{
+ CSTD_UNUSED(kbdev);
+}
+
+void kbase_job_slot_term(kbase_device *kbdev)
+{
+ int i;
+
+ OSK_ASSERT(kbdev);
+
+ for (i = 0; i < kbdev->gpu_props.num_job_slots; i++)
+ {
+ osk_spinlock_irq_term(&kbdev->jm_slots[i].lock);
+ }
+}
+KBASE_EXPORT_TEST_API(kbase_job_slot_term)
+
+
+/**
+ * Soft-stop the specified job slot
+ *
+ * The job slot lock must be held when calling this function.
+ * The job slot must not already be in the process of being soft-stopped.
+ *
+ * Where possible any job in the next register is evicted before the soft-stop.
+ *
+ * @param kbdev The kbase device
+ * @param js The job slot to soft-stop
+ * @param target_katom The job that should be soft-stopped (or NULL for any job)
+ */
+void kbase_job_slot_softstop(kbase_device *kbdev, int js, kbase_jd_atom *target_katom)
+{
+ kbasep_job_slot_soft_or_hard_stop(kbdev, NULL, js, target_katom, JSn_COMMAND_SOFT_STOP);
+}
+
+/**
+ * Hard-stop the specified job slot
+ *
+ * The job slot lock must be held when calling this function.
+ *
+ * @param kctx The kbase context that contains the job(s) that should be hard-stopped
+ * @param js The job slot to hard-stop
+ * @param target_katom The job that should be hard-stopped (or NULL for all jobs from the context)
+ */
+void kbase_job_slot_hardstop(kbase_context *kctx, int js, kbase_jd_atom *target_katom)
+{
+ kbase_device *kbdev = kctx->kbdev;
+ kbasep_job_slot_soft_or_hard_stop(kbdev, kctx, js, target_katom, JSn_COMMAND_HARD_STOP);
+
+ if (kbase_hw_has_issue(kctx->kbdev, BASE_HW_ISSUE_8401) ||
+ kbase_hw_has_issue(kctx->kbdev, BASE_HW_ISSUE_9510))
+ {
+ /* The workaround for HW issue 8401 has an issue, so instead of hard-stopping
+ * just reset the GPU. This will ensure that the jobs leave the GPU.
+ *
+ * All callers of this function immediately drop the slot lock after calling this function.
+ * So this is safe because the parent functions don't require atomicity regarding the job slot.
+ */
+ kbase_job_slot_unlock(kbdev, js);
+ if (kbase_prepare_to_reset_gpu(kbdev))
+ {
+ kbase_reset_gpu(kbdev);
+ }
+ kbase_job_slot_lock(kbdev, js);
+ }
+}
+
+void kbasep_reset_timeout_worker(osk_workq_work *data)
+{
+ kbase_device *kbdev;
+ int i;
+ kbasep_js_tick end_timestamp = kbasep_js_get_js_ticks();
+ kbasep_js_device_data *js_devdata;
+ kbase_uk_hwcnt_setup hwcnt_setup = {{0}};
+ kbase_instr_state bckp_state;
+
+ OSK_ASSERT(data);
+
+ kbdev = CONTAINER_OF(data, kbase_device, reset_work);
+
+ OSK_ASSERT(kbdev);
+
+ kbase_pm_context_active(kbdev);
+
+ js_devdata = &kbdev->js_data;
+
+ /* All slot have been soft-stopped and we've waited SOFT_STOP_RESET_TIMEOUT for the slots to clear, at this point
+ * we assume that anything that is still left on the GPU is stuck there and we'll kill it when we reset the GPU */
+
+ OSK_PRINT_ERROR(OSK_BASE_JD, "Resetting GPU");
+
+ /* Make sure the timer has completed - this cannot be done from interrupt context,
+ * so this cannot be done within kbasep_try_reset_gpu_early. */
+ osk_timer_stop(&kbdev->reset_timer);
+
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+
+ if (kbdev->hwcnt.state == KBASE_INSTR_STATE_RESETTING)
+ { /*the same interrupt handler preempted itself*/
+ /* GPU is being reset*/
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+ osk_waitq_wait(&kbdev->hwcnt.waitqueue);
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+ }
+ /* Save the HW counters setup */
+ if (kbdev->hwcnt.kctx != NULL)
+ {
+ kbase_context *kctx = kbdev->hwcnt.kctx;
+ hwcnt_setup.dump_buffer = kbase_reg_read(kbdev, GPU_CONTROL_REG(PRFCNT_BASE_LO), kctx) & 0xffffffff;
+ hwcnt_setup.dump_buffer |= (mali_addr64)kbase_reg_read(kbdev, GPU_CONTROL_REG(PRFCNT_BASE_HI), kctx) << 32;
+ hwcnt_setup.jm_bm = kbase_reg_read(kbdev, GPU_CONTROL_REG(PRFCNT_JM_EN), kctx);
+ hwcnt_setup.shader_bm = kbase_reg_read(kbdev, GPU_CONTROL_REG(PRFCNT_SHADER_EN), kctx);
+ hwcnt_setup.tiler_bm = kbase_reg_read(kbdev, GPU_CONTROL_REG(PRFCNT_TILER_EN), kctx);
+ hwcnt_setup.l3_cache_bm = kbase_reg_read(kbdev, GPU_CONTROL_REG(PRFCNT_L3_CACHE_EN), kctx);
+ hwcnt_setup.mmu_l2_bm = kbase_reg_read(kbdev, GPU_CONTROL_REG(PRFCNT_MMU_L2_EN), kctx);
+ }
+ bckp_state = kbdev->hwcnt.state;
+ kbdev->hwcnt.state = KBASE_INSTR_STATE_RESETTING;
+ osk_waitq_clear(&kbdev->hwcnt.waitqueue);
+ /* Disable IRQ to avoid IRQ handlers to kick in after releaseing the spinlock;
+ * this also clears any outstanding interrupts */
+ kbase_pm_disable_interrupts(kbdev);
+ /* Ensure that any IRQ handlers have finished */
+ kbase_synchronize_irqs(kbdev);
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+
+ /* Reset the GPU */
+ kbase_pm_power_transitioning(kbdev);
+ kbase_pm_init_hw(kbdev);
+
+ kbase_pm_power_transitioning(kbdev);
+
+ osk_spinlock_irq_lock(&kbdev->hwcnt.lock);
+ /* Restore the HW counters setup */
+ if (kbdev->hwcnt.kctx != NULL)
+ {
+ kbase_context *kctx = kbdev->hwcnt.kctx;
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_BASE_LO), hwcnt_setup.dump_buffer & 0xFFFFFFFF, kctx);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_BASE_HI), hwcnt_setup.dump_buffer >> 32, kctx);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_JM_EN), hwcnt_setup.jm_bm, kctx);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_SHADER_EN), hwcnt_setup.shader_bm, kctx);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_L3_CACHE_EN), hwcnt_setup.l3_cache_bm, kctx);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_MMU_L2_EN), hwcnt_setup.mmu_l2_bm, kctx);
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8186))
+ {
+ /* Issue 8186 requires TILER_EN to be disabled before updating PRFCNT_CONFIG. We then restore the register contents */
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_TILER_EN), 0, kctx);
+ }
+
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_CONFIG), (kctx->as_nr << PRFCNT_CONFIG_AS_SHIFT) | PRFCNT_CONFIG_MODE_MANUAL, kctx);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(PRFCNT_TILER_EN), hwcnt_setup.tiler_bm, kctx);
+ }
+ osk_waitq_set(&kbdev->hwcnt.waitqueue);
+ kbdev->hwcnt.state = bckp_state;
+ osk_spinlock_irq_unlock(&kbdev->hwcnt.lock);
+
+ /* Re-init the power policy */
+ kbase_pm_send_event(kbdev, KBASE_PM_EVENT_POLICY_INIT);
+
+ /* Wait for the policy to power up the GPU */
+ kbase_pm_wait_for_power_up(kbdev);
+
+ /* Complete any jobs that were still on the GPU */
+ for (i = 0; i < kbdev->gpu_props.num_job_slots; i++)
+ {
+ int nr_done;
+ kbase_jm_slot *slot = kbase_job_slot_lock(kbdev, i);
+
+ nr_done = kbasep_jm_nr_jobs_submitted( slot );
+ while (nr_done) {
+ OSK_PRINT_ERROR(OSK_BASE_JD, "Job stuck in slot %d on the GPU was cancelled", i);
+ kbase_job_done_slot(kbdev, i, BASE_JD_EVENT_JOB_CANCELLED, 0, &end_timestamp);
+ nr_done--;
+ }
+
+ kbase_job_slot_unlock(kbdev, i);
+ }
+
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+
+ /* Reprogram the GPU's MMU */
+ for(i = 0; i < BASE_MAX_NR_AS; i++)
+ {
+ if (js_devdata->runpool_irq.per_as_data[i].kctx) {
+ kbase_as *as = &kbdev->as[i];
+ osk_mutex_lock(&as->transaction_mutex);
+ kbase_mmu_update(js_devdata->runpool_irq.per_as_data[i].kctx);
+ osk_mutex_unlock(&as->transaction_mutex);
+ }
+ }
+
+ osk_atomic_set(&kbdev->reset_gpu, KBASE_RESET_GPU_NOT_PENDING);
+ osk_waitq_set(&kbdev->reset_waitq);
+ OSK_PRINT_ERROR(OSK_BASE_JD, "Reset complete");
+
+ /* Try submitting some jobs to restart processing */
+ KBASE_TRACE_ADD( kbdev, JM_SUBMIT_AFTER_RESET, NULL, NULL, 0u, 0 );
+ kbasep_js_try_run_next_job(kbdev);
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+
+ kbase_pm_context_idle(kbdev);
+}
+
+void kbasep_reset_timer_callback(void *data)
+{
+ kbase_device *kbdev = (kbase_device*)data;
+
+ OSK_ASSERT(kbdev);
+
+ if (osk_atomic_compare_and_swap(&kbdev->reset_gpu, KBASE_RESET_GPU_COMMITTED, KBASE_RESET_GPU_HAPPENING) !=
+ KBASE_RESET_GPU_COMMITTED)
+ {
+ /* Reset has been cancelled or has already occurred */
+ return;
+ }
+ osk_workq_submit(&kbdev->reset_workq, &kbdev->reset_work);
+}
+
+/*
+ * If all jobs are evicted from the GPU then we can reset the GPU
+ * immediately instead of waiting for the timeout to elapse
+ */
+static void kbasep_try_reset_gpu_early(kbase_device *kbdev)
+{
+ int i;
+ int pending_jobs = 0;
+
+ OSK_ASSERT(kbdev);
+
+ /* Count the number of jobs */
+ for (i = 0; i < kbdev->gpu_props.num_job_slots; i++)
+ {
+ kbase_jm_slot *slot = kbase_job_slot_lock(kbdev, i);
+ pending_jobs += kbasep_jm_nr_jobs_submitted(slot);
+ kbase_job_slot_unlock(kbdev, i);
+ }
+
+ if (pending_jobs > 0)
+ {
+ /* There are still jobs on the GPU - wait */
+ return;
+ }
+
+ /* Check that the reset has been committed to (i.e. kbase_reset_gpu has been called), and that no other
+ * thread beat this thread to starting the reset */
+ if (osk_atomic_compare_and_swap(&kbdev->reset_gpu, KBASE_RESET_GPU_COMMITTED, KBASE_RESET_GPU_HAPPENING) !=
+ KBASE_RESET_GPU_COMMITTED)
+ {
+ /* Reset has already occurred */
+ return;
+ }
+ osk_workq_submit(&kbdev->reset_workq, &kbdev->reset_work);
+}
+
+/*
+ * Prepare for resetting the GPU.
+ * This function just soft-stops all the slots to ensure that as many jobs as possible are saved.
+ *
+ * The function returns a boolean which should be interpreted as follows:
+ * - MALI_TRUE - Prepared for reset, kbase_reset_gpu should be called.
+ * - MALI_FALSE - Another thread is performing a reset, kbase_reset_gpu should not be called.
+ *
+ * @return See description
+ */
+mali_bool kbase_prepare_to_reset_gpu(kbase_device *kbdev)
+{
+ int i;
+
+ OSK_ASSERT(kbdev);
+
+ if (osk_atomic_compare_and_swap(&kbdev->reset_gpu, KBASE_RESET_GPU_NOT_PENDING, KBASE_RESET_GPU_PREPARED) !=
+ KBASE_RESET_GPU_NOT_PENDING)
+ {
+ /* Some other thread is already resetting the GPU */
+ return MALI_FALSE;
+ }
+
+ osk_waitq_clear(&kbdev->reset_waitq);
+
+ OSK_PRINT_ERROR(OSK_BASE_JD, "Preparing to soft-reset GPU: Soft-stopping all jobs");
+
+ for (i = 0; i < kbdev->gpu_props.num_job_slots; i++)
+ {
+ kbase_job_slot_lock(kbdev, i);
+ kbase_job_slot_softstop(kbdev, i, NULL);
+ kbase_job_slot_unlock(kbdev, i);
+ }
+
+ return MALI_TRUE;
+}
+
+/*
+ * This function should be called after kbase_prepare_to_reset_gpu iff it returns MALI_TRUE.
+ * It should never be called without a corresponding call to kbase_prepare_to_reset_gpu.
+ *
+ * After this function is called (or not called if kbase_prepare_to_reset_gpu returned MALI_FALSE),
+ * the caller should wait for kbdev->reset_waitq to be signalled to know when the reset has completed.
+ */
+void kbase_reset_gpu(kbase_device *kbdev)
+{
+ osk_error ret;
+ u32 timeout_ms;
+
+ OSK_ASSERT(kbdev);
+
+ /* Note this is an assert/atomic_set because it is a software issue for a race to be occuring here */
+ OSK_ASSERT(osk_atomic_get(&kbdev->reset_gpu) == KBASE_RESET_GPU_PREPARED);
+ osk_atomic_set(&kbdev->reset_gpu, KBASE_RESET_GPU_COMMITTED);
+
+ timeout_ms = kbasep_get_config_value(kbdev, kbdev->config_attributes, KBASE_CONFIG_ATTR_JS_RESET_TIMEOUT_MS);
+ ret = osk_timer_start(&kbdev->reset_timer, timeout_ms);
+ if (ret != OSK_ERR_NONE)
+ {
+ OSK_PRINT_ERROR(OSK_BASE_JD, "Failed to start timer for soft-resetting GPU");
+ /* We can't rescue jobs from the GPU so immediately reset */
+ osk_workq_submit(&kbdev->reset_workq, &kbdev->reset_work);
+ }
+
+ /* Try resetting early */
+ kbasep_try_reset_gpu_early(kbdev);
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_jm.h
+ * Job Manager Low-level APIs.
+ */
+
+#ifndef _KBASE_JM_H_
+#define _KBASE_JM_H_
+
+#include <kbase/src/common/mali_kbase_8401_workaround.h>
+#include <kbase/src/common/mali_kbase_hw.h>
+
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_kbase_api
+ * @{
+ */
+
+
+/**
+ * @addtogroup kbase_jm Job Manager Low-level APIs
+ * @{
+ *
+ */
+
+static INLINE int kbasep_jm_is_js_free(kbase_device *kbdev, int js, kbase_context *kctx)
+{
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( 0 <= js && js < kbdev->gpu_props.num_job_slots );
+
+ return !kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_COMMAND_NEXT), kctx);
+}
+
+/**
+ * This checks that:
+ * - there is enough space in the GPU's buffers (JSn_NEXT and JSn_HEAD registers) to accomodate the job.
+ * - there is enough space to track the job in a our Submit Slots. Note that we have to maintain space to
+ * requeue one job in case the next registers on the hardware need to be cleared.
+ */
+static INLINE mali_bool kbasep_jm_is_submit_slots_free(kbase_device *kbdev, int js, kbase_context *kctx)
+{
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( 0 <= js && js < kbdev->gpu_props.num_job_slots );
+
+ if (osk_atomic_get(&kbdev->reset_gpu) != KBASE_RESET_GPU_NOT_PENDING)
+ {
+ /* The GPU is being reset - so prevent submission */
+ return MALI_FALSE;
+ }
+
+ return (mali_bool)( kbasep_jm_is_js_free(kbdev, js, kctx)
+ && kbdev->jm_slots[js].submitted_nr < (BASE_JM_SUBMIT_SLOTS-2) );
+}
+
+/**
+ * Initialize a submit slot
+ */
+static INLINE void kbasep_jm_init_submit_slot( kbase_jm_slot *slot )
+{
+ slot->submitted_nr = 0;
+ slot->submitted_head = 0;
+}
+
+/**
+ * Find the atom at the idx'th element in the queue without removing it, starting at the head with idx==0.
+ */
+static INLINE kbase_jd_atom* kbasep_jm_peek_idx_submit_slot( kbase_jm_slot *slot, u8 idx )
+{
+ u8 pos;
+ kbase_jd_atom *katom;
+
+ OSK_ASSERT( idx < BASE_JM_SUBMIT_SLOTS );
+
+ pos = (slot->submitted_head + idx) & BASE_JM_SUBMIT_SLOTS_MASK;
+ katom = slot->submitted[pos];
+
+ return katom;
+}
+
+/**
+ * Pop front of the submitted
+ */
+static INLINE kbase_jd_atom* kbasep_jm_dequeue_submit_slot( kbase_jm_slot *slot )
+{
+ u8 pos;
+ kbase_jd_atom *katom;
+
+ pos = slot->submitted_head & BASE_JM_SUBMIT_SLOTS_MASK;
+ katom = slot->submitted[pos];
+ slot->submitted[pos] = NULL; /* Just to catch bugs... */
+ OSK_ASSERT(katom);
+
+ /* rotate the buffers */
+ slot->submitted_head = (slot->submitted_head + 1) & BASE_JM_SUBMIT_SLOTS_MASK;
+ slot->submitted_nr--;
+
+ OSK_PRINT_INFO( OSK_BASE_JM, "katom %p new head %u",
+ (void *)katom, (unsigned int)slot->submitted_head);
+
+ return katom;
+}
+
+/* Pop back of the submitted queue (unsubmit a job)
+ */
+static INLINE kbase_jd_atom *kbasep_jm_dequeue_tail_submit_slot( kbase_jm_slot *slot )
+{
+ u8 pos;
+
+ slot->submitted_nr--;
+
+ pos = (slot->submitted_head + slot->submitted_nr) & BASE_JM_SUBMIT_SLOTS_MASK;
+
+ return slot->submitted[pos];
+}
+
+static INLINE u8 kbasep_jm_nr_jobs_submitted( kbase_jm_slot *slot )
+{
+ return slot->submitted_nr;
+}
+
+
+/**
+ * Push back of the submitted
+ */
+static INLINE void kbasep_jm_enqueue_submit_slot( kbase_jm_slot *slot, kbase_jd_atom *katom )
+{
+ u8 nr;
+ u8 pos;
+ nr = slot->submitted_nr++;
+ OSK_ASSERT(nr < BASE_JM_SUBMIT_SLOTS);
+
+ pos = (slot->submitted_head + nr) & BASE_JM_SUBMIT_SLOTS_MASK;
+ slot->submitted[pos] = katom;
+}
+
+/**
+ * @brief Query whether a job peeked/dequeued from the submit slots is a
+ * 'dummy' job that is used for hardware workaround purposes.
+ *
+ * Any time a job is peeked/dequeued from the submit slots, this should be
+ * queried on that job.
+ *
+ * If a \a atom is indicated as being a dummy job, then you <b>must not attempt
+ * to use \a atom</b>. This is because its members will not necessarily be
+ * initialized, and so could lead to a fault if they were used.
+ *
+ * @param[in] kbdev kbase device pointer
+ * @param[in] atom The atom to query
+ *
+ * @return MALI_TRUE if \a atom is for a dummy job, in which case you must not
+ * attempt to use it.
+ * @return MALI_FALSE otherwise, and \a atom is safe to use.
+ */
+static INLINE mali_bool kbasep_jm_is_dummy_workaround_job( kbase_device *kbdev, kbase_jd_atom *atom )
+{
+ /* Query the set of workaround jobs here */
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8401))
+ {
+ if ( kbasep_8401_is_workaround_job( atom ) != MALI_FALSE )
+ {
+ return MALI_TRUE;
+ }
+ }
+
+ /* This job is not a workaround job, so it will be processed as normal */
+ return MALI_FALSE;
+}
+
+/**
+ * @brief Submit a job to a certain job-slot
+ *
+ * The caller must check kbasep_jm_is_submit_slots_free() != MALI_FALSE before calling this.
+ *
+ * The following locking conditions are made on the caller:
+ * - it must hold the kbasep_js_device_data::runpoool_irq::lock
+ * - This is to access the kbase_context::as_nr
+ * - In any case, the kbase_js code that calls this function will always have
+ * this lock held.
+ * - it must hold kbdev->jm_slots[ \a s ].lock
+ */
+void kbase_job_submit_nolock(kbase_device *kbdev, kbase_jd_atom *katom, int js);
+
+/**
+ * @brief Complete the head job on a particular job-slot
+ */
+void kbase_job_done_slot(kbase_device *kbdev, int s, u32 completion_code, u64 job_tail, kbasep_js_tick *end_timestamp);
+
+/**
+ * @brief Obtain the lock for a job slot.
+ *
+ * This function also returns the structure for the specified job slot to simplify the code
+ *
+ * @param[in] kbdev Kbase device pointer
+ * @param[in] js The job slot number to lock
+ *
+ * @return The job slot structure
+ */
+static INLINE kbase_jm_slot *kbase_job_slot_lock(kbase_device *kbdev, int js)
+{
+ osk_spinlock_irq_lock(&kbdev->jm_slots[js].lock);
+ return &kbdev->jm_slots[js];
+}
+
+/**
+ * @brief Release the lock for a job slot
+ *
+ * @param[in] kbdev Kbase device pointer
+ * @param[in] js The job slot number to unlock
+ */
+static INLINE void kbase_job_slot_unlock(kbase_device *kbdev, int js)
+{
+ osk_spinlock_irq_unlock(&kbdev->jm_slots[js].lock);
+}
+
+/** @} */ /* end group kbase_jm */
+/** @} */ /* end group base_kbase_api */
+/** @} */ /* end group base_api */
+
+#endif /* _KBASE_JM_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/*
+ * Job Scheduler Implementation
+ */
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_kbase_js.h>
+#include <kbase/src/common/mali_kbase_js_affinity.h>
+#include <kbase/src/common/mali_kbase_gator.h>
+#include <kbase/src/common/mali_kbase_hw.h>
+
+#include "mali_kbase_jm.h"
+#include <kbase/src/common/mali_kbase_defs.h>
+
+/*
+ * Private types
+ */
+
+/** Bitpattern indicating the result of releasing a context */
+enum
+{
+ /** The context was descheduled - caller should try scheduling in a new one
+ * to keep the runpool full */
+ KBASEP_JS_RELEASE_RESULT_WAS_DESCHEDULED = (1u << 0),
+
+ /** The Runpool's context attributes changed. The scheduler might be able to
+ * submit more jobs than previously, and so the caller should call
+ * kbasep_js_try_run_next_job(). */
+ KBASEP_JS_RELEASE_RESULT_CTX_ATTR_CHANGE = (1u << 1)
+
+};
+
+typedef u32 kbasep_js_release_result;
+
+/*
+ * Private function prototypes
+ */
+STATIC INLINE void kbasep_js_deref_permon_check_and_disable_cycle_counter( kbase_device *kbdev,
+ kbase_jd_atom * katom );
+
+STATIC INLINE void kbasep_js_ref_permon_check_and_enable_cycle_counter( kbase_device *kbdev,
+ kbase_jd_atom * katom );
+
+STATIC kbasep_js_release_result kbasep_js_runpool_release_ctx_internal( kbase_device *kbdev, kbase_context *kctx );
+
+/** Helper for trace subcodes */
+#if KBASE_TRACE_ENABLE != 0
+STATIC int kbasep_js_trace_get_refcnt( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ int as_nr;
+ int refcnt = 0;
+
+ js_devdata = &kbdev->js_data;
+
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+ as_nr = kctx->as_nr;
+ if ( as_nr != KBASEP_AS_NR_INVALID )
+ {
+ kbasep_js_per_as_data *js_per_as_data;
+ js_per_as_data = &js_devdata->runpool_irq.per_as_data[as_nr];
+
+ refcnt = js_per_as_data->as_busy_refcount;
+ }
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+
+ return refcnt;
+}
+#else /* KBASE_TRACE_ENABLE != 0 */
+STATIC int kbasep_js_trace_get_refcnt( kbase_device *kbdev, kbase_context *kctx )
+{
+ CSTD_UNUSED( kbdev );
+ CSTD_UNUSED( kctx );
+ return 0;
+}
+#endif /* KBASE_TRACE_ENABLE != 0 */
+
+
+
+/*
+ * Private types
+ */
+enum
+{
+ JS_DEVDATA_INIT_NONE =0,
+ JS_DEVDATA_INIT_CONSTANTS =(1 << 0),
+ JS_DEVDATA_INIT_RUNPOOL_MUTEX =(1 << 1),
+ JS_DEVDATA_INIT_QUEUE_MUTEX =(1 << 2),
+ JS_DEVDATA_INIT_RUNPOOL_IRQ_LOCK=(1 << 3),
+ JS_DEVDATA_INIT_POLICY =(1 << 4),
+ JS_DEVDATA_INIT_ALL =((1 << 5)-1)
+};
+
+enum
+{
+ JS_KCTX_INIT_NONE =0,
+ JS_KCTX_INIT_CONSTANTS =(1 << 0),
+ JS_KCTX_INIT_JSCTX_MUTEX =(1 << 1),
+ JS_KCTX_INIT_POLICY =(1 << 2),
+ JS_KCTX_INIT_JSCTX_WAITQ_SCHED =(1 << 3),
+ JS_KCTX_INIT_JSCTX_WAITQ_NSCHED =(1 << 4),
+ JS_KCTX_INIT_ALL =((1 << 5)-1)
+};
+
+/*
+ * Private functions
+ */
+
+/**
+ * Check if the job had performance monitoring enabled and decrement the count. If no jobs require
+ * performance monitoring, then the cycle counters will be disabled in the GPU.
+ *
+ * No locks need to be held - locking is handled further down
+ *
+ * This function does not sleep.
+ */
+
+STATIC INLINE void kbasep_js_deref_permon_check_and_disable_cycle_counter( kbase_device *kbdev, kbase_jd_atom * katom )
+{
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( katom != NULL );
+
+ if ( katom->core_req & BASE_JD_REQ_PERMON )
+ {
+ kbase_pm_release_gpu_cycle_counter(kbdev);
+ }
+}
+
+/**
+ * Check if the job has performance monitoring enabled and keep a count of it. If at least one
+ * job requires performance monitoring, then the cycle counters will be enabled in the GPU.
+ *
+ * No locks need to be held - locking is handled further down
+ *
+ * This function does not sleep.
+ */
+
+STATIC INLINE void kbasep_js_ref_permon_check_and_enable_cycle_counter( kbase_device *kbdev, kbase_jd_atom * katom )
+{
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( katom != NULL );
+
+ if ( katom->core_req & BASE_JD_REQ_PERMON )
+ {
+ kbase_pm_request_gpu_cycle_counter(kbdev);
+ }
+}
+
+/*
+ * The following locking conditions are made on the caller:
+ * - The caller must hold the kbasep_js_kctx_info::ctx::jsctx_mutex.
+ * - The caller must hold the kbasep_js_device_data::runpool_mutex
+ */
+STATIC INLINE void runpool_inc_context_count( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_kctx_info *js_kctx_info;
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ js_devdata = &kbdev->js_data;
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ /* Track total contexts */
+ ++(js_devdata->nr_all_contexts_running);
+
+ if ( (js_kctx_info->ctx.flags & KBASE_CTX_FLAG_SUBMIT_DISABLED) == 0 )
+ {
+ /* Track contexts that can submit jobs */
+ ++(js_devdata->nr_user_contexts_running);
+ }
+}
+
+/*
+ * The following locking conditions are made on the caller:
+ * - The caller must hold the kbasep_js_kctx_info::ctx::jsctx_mutex.
+ * - The caller must hold the kbasep_js_device_data::runpool_mutex
+ */
+STATIC INLINE void runpool_dec_context_count( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_kctx_info *js_kctx_info;
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ js_devdata = &kbdev->js_data;
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ /* Track total contexts */
+ --(js_devdata->nr_all_contexts_running);
+
+ if ( (js_kctx_info->ctx.flags & KBASE_CTX_FLAG_SUBMIT_DISABLED) == 0 )
+ {
+ /* Track contexts that can submit jobs */
+ --(js_devdata->nr_user_contexts_running);
+ }
+}
+
+/**
+ * @brief check whether the runpool is full for a specified context
+ *
+ * If kctx == NULL, then this makes the least restrictive check on the
+ * runpool. A specific context that is supplied immediately after could fail
+ * the check, even under the same conditions.
+ *
+ * Therefore, once a context is obtained you \b must re-check it with this
+ * function, since the return value could change to MALI_FALSE.
+ *
+ * The following locking conditions are made on the caller:
+ * - In all cases, the caller must hold kbasep_js_device_data::runpool_mutex
+ * - When kctx != NULL the caller must hold the kbasep_js_kctx_info::ctx::jsctx_mutex.
+ * - When kctx == NULL, then the caller need not hold any jsctx_mutex locks (but it doesn't do any harm to do so).
+ */
+STATIC mali_bool check_is_runpool_full( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ mali_bool is_runpool_full;
+ OSK_ASSERT( kbdev != NULL );
+
+ js_devdata = &kbdev->js_data;
+
+ is_runpool_full = (mali_bool)(js_devdata->nr_all_contexts_running >= kbdev->nr_hw_address_spaces);
+
+ if ( kctx != NULL && (kctx->jctx.sched_info.ctx.flags & KBASE_CTX_FLAG_SUBMIT_DISABLED) == 0 )
+ {
+ /* Contexts that don't submit might use less of the address spaces available, due to HW workarounds */
+ is_runpool_full = (mali_bool)(js_devdata->nr_user_contexts_running >= kbdev->nr_user_address_spaces);
+ }
+
+ return is_runpool_full;
+}
+
+
+STATIC base_jd_core_req core_reqs_from_jsn_features( u16 features /* JS<n>_FEATURE register value */ )
+{
+ base_jd_core_req core_req = 0u;
+
+ if ( (features & JSn_FEATURE_SET_VALUE_JOB) != 0 )
+ {
+ core_req |= BASE_JD_REQ_V;
+ }
+ if ( (features & JSn_FEATURE_CACHE_FLUSH_JOB) != 0 )
+ {
+ core_req |= BASE_JD_REQ_CF;
+ }
+ if ( (features & JSn_FEATURE_COMPUTE_JOB) != 0 )
+ {
+ core_req |= BASE_JD_REQ_CS;
+ }
+ if ( (features & JSn_FEATURE_TILER_JOB) != 0 )
+ {
+ core_req |= BASE_JD_REQ_T;
+ }
+ if ( (features & JSn_FEATURE_FRAGMENT_JOB) != 0 )
+ {
+ core_req |= BASE_JD_REQ_FS;
+ }
+ return core_req;
+}
+
+/**
+ * Picks a free address space and add the context to the Policy. Then perform a
+ * transaction on this AS and RunPool IRQ to:
+ * - setup the runpool_irq structure and the context on that AS
+ * - Activate the MMU on that AS
+ * - Allow jobs to be submitted on that AS
+ *
+ * Locking conditions:
+ * - Caller must hold the kbasep_js_kctx_info::jsctx_mutex
+ * - Caller must hold the kbase_js_device_data::runpool_mutex
+ * - AS transaction mutex will be obtained
+ * - Runpool IRQ lock will be obtained
+ */
+STATIC void assign_and_activate_kctx_addr_space( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ kbase_as *current_as;
+ kbasep_js_per_as_data *js_per_as_data;
+ long ffs_result;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ js_devdata = &kbdev->js_data;
+
+ /* Find the free address space */
+ ffs_result = osk_find_first_set_bit( js_devdata->as_free );
+ /* ASSERT that we should've found a free one */
+ OSK_ASSERT( 0 <= ffs_result && ffs_result < kbdev->nr_hw_address_spaces );
+ js_devdata->as_free &= ~((u16)(1u << ffs_result));
+
+ /*
+ * Transaction on the AS and runpool_irq
+ */
+ current_as = &kbdev->as[ffs_result];
+ js_per_as_data = &js_devdata->runpool_irq.per_as_data[ffs_result];
+ osk_mutex_lock( ¤t_as->transaction_mutex );
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+
+ /* Attribute handling */
+ kbasep_js_ctx_attr_runpool_retain_ctx( kbdev, kctx );
+
+ /* Assign addr space */
+ kctx->as_nr = (int)ffs_result;
+#if MALI_GATOR_SUPPORT
+ kbase_trace_mali_mmu_as_in_use(kctx->as_nr);
+#endif
+ /* Activate this address space on the MMU */
+ kbase_mmu_update( kctx );
+
+ /* Allow it to run jobs */
+ kbasep_js_set_submit_allowed( js_devdata, kctx );
+
+ /* Book-keeping */
+ js_per_as_data->kctx = kctx;
+ js_per_as_data->as_busy_refcount = 0;
+
+ /* Lastly, add the context to the policy's runpool - this really allows it to run jobs */
+ kbasep_js_policy_runpool_add_ctx( &js_devdata->policy, kctx );
+ /*
+ * Transaction complete
+ */
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+ osk_mutex_unlock( ¤t_as->transaction_mutex );
+
+}
+
+void kbasep_js_try_run_next_job( kbase_device *kbdev )
+{
+ int js;
+
+ OSK_ASSERT( kbdev != NULL );
+
+ for ( js = 0; js < kbdev->gpu_props.num_job_slots ; ++js )
+ {
+ kbasep_js_try_run_next_job_on_slot( kbdev, js );
+ }
+}
+
+/** Hold the kbasep_js_device_data::runpool_irq::lock for this */
+mali_bool kbasep_js_runpool_retain_ctx_nolock( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_per_as_data *js_per_as_data;
+ mali_bool result = MALI_FALSE;
+ int as_nr;
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+ js_devdata = &kbdev->js_data;
+
+ as_nr = kctx->as_nr;
+ if ( as_nr != KBASEP_AS_NR_INVALID )
+ {
+ int new_refcnt;
+
+ OSK_ASSERT( as_nr >= 0 );
+ js_per_as_data = &js_devdata->runpool_irq.per_as_data[as_nr];
+
+ OSK_ASSERT( js_per_as_data->kctx != NULL );
+
+ new_refcnt = ++(js_per_as_data->as_busy_refcount);
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, JS_RETAIN_CTX_NOLOCK, kctx, NULL, 0u,
+ new_refcnt );
+ result = MALI_TRUE;
+ }
+
+ return result;
+}
+
+/*
+ * Functions private to KBase ('Protected' functions)
+ */
+void kbase_js_try_run_jobs( kbase_device *kbdev )
+{
+ kbasep_js_device_data *js_devdata;
+
+ OSK_ASSERT( kbdev != NULL );
+ js_devdata = &kbdev->js_data;
+
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+
+ kbasep_js_try_run_next_job( kbdev );
+
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+}
+
+mali_error kbasep_js_devdata_init( kbase_device *kbdev )
+{
+ kbasep_js_device_data *js_devdata;
+ mali_error err;
+ int i;
+ u16 as_present;
+ osk_error osk_err;
+
+ OSK_ASSERT( kbdev != NULL );
+
+ js_devdata = &kbdev->js_data;
+
+ OSK_ASSERT( js_devdata->init_status == JS_DEVDATA_INIT_NONE );
+
+ /* These two must be recalculated if nr_hw_address_spaces changes (e.g. for HW workarounds) */
+ as_present = (1U << kbdev->nr_hw_address_spaces) - 1;
+ kbdev->nr_user_address_spaces = kbdev->nr_hw_address_spaces;
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8987))
+ {
+ mali_bool use_workaround_for_security;
+ use_workaround_for_security = (mali_bool)kbasep_get_config_value( kbdev, kbdev->config_attributes, KBASE_CONFIG_ATTR_SECURE_BUT_LOSS_OF_PERFORMANCE );
+ if ( use_workaround_for_security != MALI_FALSE )
+ {
+ OSK_PRINT(OSK_BASE_JM, "GPU has HW ISSUE 8987, and driver configured for security workaround: 1 address space only");
+ kbdev->nr_user_address_spaces = 1;
+ }
+ }
+#if MALI_DEBUG
+ /* Soft-stop will be disabled on a single context by default unless softstop_always is set */
+ js_devdata->softstop_always = MALI_FALSE;
+#endif /* MALI_DEBUG */
+ js_devdata->nr_all_contexts_running = 0;
+ js_devdata->nr_user_contexts_running = 0;
+ js_devdata->as_free = as_present; /* All ASs initially free */
+ js_devdata->runpool_irq.submit_allowed = 0u; /* No ctx allowed to submit */
+ OSK_MEMSET( js_devdata->runpool_irq.ctx_attr_ref_count, 0, sizeof(js_devdata->runpool_irq.ctx_attr_ref_count) );
+ OSK_MEMSET( js_devdata->runpool_irq.slot_affinities, 0, sizeof( js_devdata->runpool_irq.slot_affinities ) );
+ js_devdata->runpool_irq.slots_blocked_on_affinity = 0u;
+ OSK_MEMSET( js_devdata->runpool_irq.slot_affinity_refcount, 0, sizeof( js_devdata->runpool_irq.slot_affinity_refcount ) );
+
+ /* Config attributes */
+ js_devdata->scheduling_tick_ns = (u32)kbasep_get_config_value( kbdev, kbdev->config_attributes, KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS );
+ js_devdata->soft_stop_ticks = (u32)kbasep_get_config_value( kbdev, kbdev->config_attributes, KBASE_CONFIG_ATTR_JS_SOFT_STOP_TICKS );
+ js_devdata->hard_stop_ticks_ss = (u32)kbasep_get_config_value( kbdev, kbdev->config_attributes, KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS );
+ js_devdata->hard_stop_ticks_nss = (u32)kbasep_get_config_value( kbdev, kbdev->config_attributes, KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS );
+ js_devdata->gpu_reset_ticks_ss = (u32)kbasep_get_config_value( kbdev, kbdev->config_attributes, KBASE_CONFIG_ATTR_JS_RESET_TICKS_SS );
+ js_devdata->gpu_reset_ticks_nss = (u32)kbasep_get_config_value( kbdev, kbdev->config_attributes, KBASE_CONFIG_ATTR_JS_RESET_TICKS_NSS );
+ js_devdata->ctx_timeslice_ns = (u32)kbasep_get_config_value( kbdev, kbdev->config_attributes, KBASE_CONFIG_ATTR_JS_CTX_TIMESLICE_NS );
+ js_devdata->cfs_ctx_runtime_init_slices = (u32)kbasep_get_config_value( kbdev, kbdev->config_attributes, KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_INIT_SLICES );
+ js_devdata->cfs_ctx_runtime_min_slices = (u32)kbasep_get_config_value( kbdev, kbdev->config_attributes, KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_MIN_SLICES );
+
+ OSK_PRINT_INFO( OSK_BASE_JM, "JS Config Attribs: " );
+ OSK_PRINT_INFO( OSK_BASE_JM, "\tjs_devdata->scheduling_tick_ns:%u", js_devdata->scheduling_tick_ns );
+ OSK_PRINT_INFO( OSK_BASE_JM, "\tjs_devdata->soft_stop_ticks:%u", js_devdata->soft_stop_ticks );
+ OSK_PRINT_INFO( OSK_BASE_JM, "\tjs_devdata->hard_stop_ticks_ss:%u", js_devdata->hard_stop_ticks_ss );
+ OSK_PRINT_INFO( OSK_BASE_JM, "\tjs_devdata->hard_stop_ticks_nss:%u", js_devdata->hard_stop_ticks_nss );
+ OSK_PRINT_INFO( OSK_BASE_JM, "\tjs_devdata->gpu_reset_ticks_ss:%u", js_devdata->gpu_reset_ticks_ss );
+ OSK_PRINT_INFO( OSK_BASE_JM, "\tjs_devdata->gpu_reset_ticks_nss:%u", js_devdata->gpu_reset_ticks_nss );
+ OSK_PRINT_INFO( OSK_BASE_JM, "\tjs_devdata->ctx_timeslice_ns:%u", js_devdata->ctx_timeslice_ns );
+ OSK_PRINT_INFO( OSK_BASE_JM, "\tjs_devdata->cfs_ctx_runtime_init_slices:%u", js_devdata->cfs_ctx_runtime_init_slices );
+ OSK_PRINT_INFO( OSK_BASE_JM, "\tjs_devdata->cfs_ctx_runtime_min_slices:%u", js_devdata->cfs_ctx_runtime_min_slices );
+
+#if MALI_BACKEND_KERNEL /* Only output on real kernel modules, otherwise it fills up multictx testing output */
+#if KBASE_DISABLE_SCHEDULING_SOFT_STOPS != 0
+ OSK_PRINT( OSK_BASE_JM,
+ "Job Scheduling Policy Soft-stops disabled, ignoring value for soft_stop_ticks==%u at %uns per tick. Other soft-stops may still occur.",
+ js_devdata->soft_stop_ticks,
+ js_devdata->scheduling_tick_ns );
+#endif
+#if KBASE_DISABLE_SCHEDULING_HARD_STOPS != 0
+ OSK_PRINT( OSK_BASE_JM,
+ "Job Scheduling Policy Hard-stops disabled, ignoring values for hard_stop_ticks_ss==%d and hard_stop_ticks_nss==%u at %uns per tick. Other hard-stops may still occur.",
+ js_devdata->hard_stop_ticks_ss,
+ js_devdata->hard_stop_ticks_nss,
+ js_devdata->scheduling_tick_ns );
+#endif
+#if KBASE_DISABLE_SCHEDULING_SOFT_STOPS != 0 && KBASE_DISABLE_SCHEDULING_HARD_STOPS != 0
+ OSK_PRINT( OSK_BASE_JM, "Note: The JS policy's tick timer (if coded) will still be run, but do nothing." );
+#endif
+#endif /* MALI_BACKEND_KERNEL */
+
+ /* setup the number of irq throttle cycles base on given time */
+ {
+ u32 irq_throttle_time_us = kbdev->gpu_props.irq_throttle_time_us;
+ u32 irq_throttle_cycles = kbasep_js_convert_us_to_gpu_ticks_max_freq(kbdev, irq_throttle_time_us);
+ osk_atomic_set( &kbdev->irq_throttle_cycles, irq_throttle_cycles);
+ }
+
+ /* Clear the AS data, including setting NULL pointers */
+ OSK_MEMSET( &js_devdata->runpool_irq.per_as_data[0], 0, sizeof(js_devdata->runpool_irq.per_as_data) );
+
+ for ( i = 0; i < kbdev->gpu_props.num_job_slots; ++i )
+ {
+ js_devdata->js_reqs[i] = core_reqs_from_jsn_features( kbdev->gpu_props.props.raw_props.js_features[i] );
+ }
+ js_devdata->init_status |= JS_DEVDATA_INIT_CONSTANTS;
+
+ /* On error, we could continue on: providing none of the below resources
+ * rely on the ones above */
+
+ osk_err = osk_mutex_init( &js_devdata->runpool_mutex, OSK_LOCK_ORDER_JS_RUNPOOL );
+ if ( osk_err == OSK_ERR_NONE )
+ {
+ js_devdata->init_status |= JS_DEVDATA_INIT_RUNPOOL_MUTEX;
+ }
+
+ osk_err = osk_mutex_init( &js_devdata->queue_mutex, OSK_LOCK_ORDER_JS_QUEUE );
+ if ( osk_err == OSK_ERR_NONE )
+ {
+ js_devdata->init_status |= JS_DEVDATA_INIT_QUEUE_MUTEX;
+ }
+
+ osk_err = osk_spinlock_irq_init( &js_devdata->runpool_irq.lock, OSK_LOCK_ORDER_JS_RUNPOOL_IRQ );
+ if ( osk_err == OSK_ERR_NONE )
+ {
+ js_devdata->init_status |= JS_DEVDATA_INIT_RUNPOOL_IRQ_LOCK;
+ }
+
+ err = kbasep_js_policy_init( kbdev );
+ if ( err == MALI_ERROR_NONE)
+ {
+ js_devdata->init_status |= JS_DEVDATA_INIT_POLICY;
+ }
+
+ /* On error, do no cleanup; this will be handled by the caller(s), since
+ * we've designed this resource to be safe to terminate on init-fail */
+ if ( js_devdata->init_status != JS_DEVDATA_INIT_ALL)
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ return MALI_ERROR_NONE;
+}
+
+void kbasep_js_devdata_halt( kbase_device *kbdev )
+{
+ CSTD_UNUSED(kbdev);
+}
+
+void kbasep_js_devdata_term( kbase_device *kbdev )
+{
+ kbasep_js_device_data *js_devdata;
+
+ OSK_ASSERT( kbdev != NULL );
+
+ js_devdata = &kbdev->js_data;
+
+ if ( (js_devdata->init_status & JS_DEVDATA_INIT_CONSTANTS) )
+ {
+ s8 zero_ctx_attr_ref_count[KBASEP_JS_CTX_ATTR_COUNT] = { 0, };
+ /* The caller must de-register all contexts before calling this */
+ OSK_ASSERT( js_devdata->nr_all_contexts_running == 0 );
+ OSK_ASSERT( OSK_MEMCMP( js_devdata->runpool_irq.ctx_attr_ref_count, zero_ctx_attr_ref_count, sizeof(js_devdata->runpool_irq.ctx_attr_ref_count)) == 0 );
+ CSTD_UNUSED( zero_ctx_attr_ref_count );
+ }
+ if ( (js_devdata->init_status & JS_DEVDATA_INIT_POLICY) )
+ {
+ kbasep_js_policy_term( &js_devdata->policy );
+ }
+ if ( (js_devdata->init_status & JS_DEVDATA_INIT_RUNPOOL_IRQ_LOCK) )
+ {
+ osk_spinlock_irq_term( &js_devdata->runpool_irq.lock );
+ }
+ if ( (js_devdata->init_status & JS_DEVDATA_INIT_QUEUE_MUTEX) )
+ {
+ osk_mutex_term( &js_devdata->queue_mutex );
+ }
+ if ( (js_devdata->init_status & JS_DEVDATA_INIT_RUNPOOL_MUTEX) )
+ {
+ osk_mutex_term( &js_devdata->runpool_mutex );
+ }
+
+ js_devdata->init_status = JS_DEVDATA_INIT_NONE;
+}
+
+
+mali_error kbasep_js_kctx_init( kbase_context *kctx )
+{
+ kbase_device *kbdev;
+ kbasep_js_kctx_info *js_kctx_info;
+ mali_error err;
+ osk_error osk_err;
+
+ OSK_ASSERT( kctx != NULL );
+
+ kbdev = kctx->kbdev;
+ OSK_ASSERT( kbdev != NULL );
+
+ js_kctx_info = &kctx->jctx.sched_info;
+ OSK_ASSERT( js_kctx_info->init_status == JS_KCTX_INIT_NONE );
+
+ js_kctx_info->ctx.nr_jobs = 0;
+ js_kctx_info->ctx.is_scheduled = MALI_FALSE;
+ js_kctx_info->ctx.is_dying = MALI_FALSE;
+ OSK_MEMSET( js_kctx_info->ctx.ctx_attr_ref_count, 0, sizeof(js_kctx_info->ctx.ctx_attr_ref_count) );
+
+ /* Initially, the context is disabled from submission until the create flags are set */
+ js_kctx_info->ctx.flags = KBASE_CTX_FLAG_SUBMIT_DISABLED;
+
+ js_kctx_info->init_status |= JS_KCTX_INIT_CONSTANTS;
+
+ /* On error, we could continue on: providing none of the below resources
+ * rely on the ones above */
+ osk_err = osk_mutex_init( &js_kctx_info->ctx.jsctx_mutex, OSK_LOCK_ORDER_JS_CTX );
+ if ( osk_err == OSK_ERR_NONE )
+ {
+ js_kctx_info->init_status |= JS_KCTX_INIT_JSCTX_MUTEX;
+ }
+
+ osk_err = osk_waitq_init( &js_kctx_info->ctx.scheduled_waitq );
+ if ( osk_err == OSK_ERR_NONE )
+ {
+ js_kctx_info->init_status |= JS_KCTX_INIT_JSCTX_WAITQ_SCHED;
+ }
+
+ osk_err = osk_waitq_init( &js_kctx_info->ctx.not_scheduled_waitq );
+ if ( osk_err == OSK_ERR_NONE )
+ {
+ js_kctx_info->init_status |= JS_KCTX_INIT_JSCTX_WAITQ_NSCHED;
+ }
+
+ err = kbasep_js_policy_init_ctx( kbdev, kctx );
+ if ( err == MALI_ERROR_NONE )
+ {
+ js_kctx_info->init_status |= JS_KCTX_INIT_POLICY;
+ }
+
+ /* On error, do no cleanup; this will be handled by the caller(s), since
+ * we've designed this resource to be safe to terminate on init-fail */
+ if ( js_kctx_info->init_status != JS_KCTX_INIT_ALL)
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ /* Initially, the context is not scheduled */
+ osk_waitq_clear( &js_kctx_info->ctx.scheduled_waitq );
+ osk_waitq_set( &js_kctx_info->ctx.not_scheduled_waitq );
+
+ return MALI_ERROR_NONE;
+}
+
+void kbasep_js_kctx_term( kbase_context *kctx )
+{
+ kbase_device *kbdev;
+ kbasep_js_kctx_info *js_kctx_info;
+ kbasep_js_policy *js_policy;
+
+ OSK_ASSERT( kctx != NULL );
+
+ kbdev = kctx->kbdev;
+ OSK_ASSERT( kbdev != NULL );
+
+ js_policy = &kbdev->js_data.policy;
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ if ( (js_kctx_info->init_status & JS_KCTX_INIT_CONSTANTS) )
+ {
+ /* The caller must de-register all jobs before calling this */
+ OSK_ASSERT( js_kctx_info->ctx.is_scheduled == MALI_FALSE );
+ OSK_ASSERT( js_kctx_info->ctx.nr_jobs == 0 );
+ /* Only certain Ctx Attributes will be zero (others can have a non-zero value for the life of the context) */
+ OSK_ASSERT( kbasep_js_ctx_attr_count_on_runpool( kbdev, KBASEP_JS_CTX_ATTR_NSS ) == 0 );
+ }
+
+ if ( (js_kctx_info->init_status & JS_KCTX_INIT_JSCTX_WAITQ_SCHED) )
+ {
+ osk_waitq_term( &js_kctx_info->ctx.scheduled_waitq );
+ }
+
+ if ( (js_kctx_info->init_status & JS_KCTX_INIT_JSCTX_WAITQ_NSCHED) )
+ {
+ osk_waitq_term( &js_kctx_info->ctx.not_scheduled_waitq );
+ }
+
+ if ( (js_kctx_info->init_status & JS_KCTX_INIT_JSCTX_MUTEX) )
+ {
+ osk_mutex_term( &js_kctx_info->ctx.jsctx_mutex );
+ }
+
+ if ( (js_kctx_info->init_status & JS_KCTX_INIT_POLICY) )
+ {
+ kbasep_js_policy_term_ctx( js_policy, kctx );
+ }
+
+ js_kctx_info->init_status = JS_KCTX_INIT_NONE;
+}
+
+/* Evict jobs from the NEXT registers
+ *
+ * The caller must hold:
+ * - kbasep_js_kctx_info::ctx::jsctx_mutex
+ * - kbasep_js_device_data::runpool_mutex
+ */
+STATIC void kbasep_js_runpool_evict_next_jobs( kbase_device *kbdev )
+{
+ int js;
+ kbasep_js_device_data *js_devdata;
+ u16 saved_submit_mask;
+
+ js_devdata = &kbdev->js_data;
+
+ /* Prevent contexts in the runpool from submitting jobs */
+ osk_spinlock_irq_lock(&js_devdata->runpool_irq.lock);
+ {
+ saved_submit_mask = js_devdata->runpool_irq.submit_allowed;
+ js_devdata->runpool_irq.submit_allowed = 0;
+ }
+ osk_spinlock_irq_unlock(&js_devdata->runpool_irq.lock);
+
+ /* Evict jobs from the NEXT registers */
+ for (js = 0; js < kbdev->gpu_props.num_job_slots; js++)
+ {
+ kbase_jm_slot *slot;
+ kbase_jd_atom *tail;
+
+ kbase_job_slot_lock(kbdev, js);
+
+ if (!kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_COMMAND_NEXT), NULL))
+ {
+ /* No job in the NEXT register */
+ kbase_job_slot_unlock(kbdev, js);
+ continue;
+ }
+
+ slot = &kbdev->jm_slots[js];
+ tail = kbasep_jm_peek_idx_submit_slot(slot, slot->submitted_nr-1);
+
+ /* Clearing job from next registers */
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_COMMAND_NEXT), JSn_COMMAND_NOP, NULL);
+
+ /* Check to see if we did remove a job from the next registers */
+ if (kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_LO), NULL) != 0 ||
+ kbase_reg_read(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_HI), NULL) != 0)
+ {
+ /* The job was successfully cleared from the next registers, requeue it */
+ slot->submitted_nr--;
+
+ /* Set the next registers to NULL */
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_LO), 0, NULL);
+ kbase_reg_write(kbdev, JOB_SLOT_REG(js, JSn_HEAD_NEXT_HI), 0, NULL);
+
+ tail->event.event_code = BASE_JD_EVENT_REMOVED_FROM_NEXT;
+
+ /* Complete the job, indicate that it took no time, and start_new_jobs==MALI_FALSE */
+ kbase_jd_done(tail, js, NULL, MALI_FALSE);
+ }
+
+ kbase_job_slot_unlock(kbdev, js);
+ }
+
+ /* Allow contexts in the runpool to submit jobs again */
+ osk_spinlock_irq_lock(&js_devdata->runpool_irq.lock);
+ {
+ js_devdata->runpool_irq.submit_allowed = saved_submit_mask;
+ }
+ osk_spinlock_irq_unlock(&js_devdata->runpool_irq.lock);
+}
+
+/**
+ * Fast start a higher priority job
+ * If the runpool is full, the lower priority contexts with no running jobs
+ * will be evicted from the runpool
+ *
+ * If \a kctx_new is NULL, the first context with no running jobs will be evicted
+ *
+ * The following locking conditions are made on the caller:
+ * - The caller must \b not hold \a kctx_new's
+ * kbasep_js_kctx_info::ctx::jsctx_mutex, or that mutex of any ctx in the
+ * runpool. This is because \a kctx_new's jsctx_mutex and one of the other
+ * scheduled ctx's jsctx_mutex will be obtained internally.
+ * - it must \em not hold kbasep_js_device_data::runpool_irq::lock (as this will be
+ * obtained internally)
+ * - it must \em not hold kbasep_js_device_data::runpool_mutex (as this will be
+ * obtained internally)
+ * - it must \em not hold kbasep_jd_device_data::queue_mutex (again, it's used
+ * internally).
+ */
+STATIC void kbasep_js_runpool_attempt_fast_start_ctx( kbase_device *kbdev, kbase_context *kctx_new )
+{
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_kctx_info *js_kctx_new;
+ kbasep_js_policy *js_policy;
+ kbasep_js_per_as_data *js_per_as_data;
+ int evict_as_nr;
+
+ OSK_ASSERT(kbdev != NULL);
+
+ js_devdata = &kbdev->js_data;
+ js_policy = &kbdev->js_data.policy;
+
+ if (kctx_new != NULL)
+ {
+ js_kctx_new = &kctx_new->jctx.sched_info;
+ osk_mutex_lock( &js_kctx_new->ctx.jsctx_mutex );
+ }
+ else
+ {
+ js_kctx_new = NULL;
+ CSTD_UNUSED(js_kctx_new);
+ }
+
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+
+ /* If the runpool is full, attempt to fast start our context */
+ if (check_is_runpool_full(kbdev, kctx_new) != MALI_FALSE)
+ {
+ /* No free address spaces - attempt to evict non-running lower priority context */
+ osk_spinlock_irq_lock(&js_devdata->runpool_irq.lock);
+ for(evict_as_nr = 0; evict_as_nr < kbdev->nr_hw_address_spaces; evict_as_nr++)
+ {
+ kbase_context *kctx_evict;
+ js_per_as_data = &js_devdata->runpool_irq.per_as_data[evict_as_nr];
+ kctx_evict = js_per_as_data->kctx;
+
+ /* Look for the AS which is not currently running */
+ if(0 == js_per_as_data->as_busy_refcount && kctx_evict != NULL)
+ {
+ /* Now compare the scheduled priority we are considering evicting with the new ctx priority
+ * and take into consideration if the scheduled priority is a realtime policy or not.
+ * Note that the lower the number, the higher the priority
+ */
+ if((kctx_new == NULL) || kbasep_js_policy_ctx_has_priority(js_policy, kctx_evict, kctx_new))
+ {
+ mali_bool retain_result;
+ kbasep_js_release_result release_result;
+ KBASE_TRACE_ADD( kbdev, JS_FAST_START_EVICTS_CTX, kctx_evict, NULL, 0u, (u32)kctx_new );
+
+ /* Retain the ctx to work on it - this shouldn't be able to fail */
+ retain_result = kbasep_js_runpool_retain_ctx_nolock( kbdev, kctx_evict );
+ OSK_ASSERT( retain_result != MALI_FALSE );
+ CSTD_UNUSED( retain_result );
+
+ /* This will cause the context to be scheduled out on the next runpool_release_ctx(),
+ * and also stop its refcount increasing */
+ kbasep_js_clear_submit_allowed(js_devdata, kctx_evict);
+
+ osk_spinlock_irq_unlock(&js_devdata->runpool_irq.lock);
+ osk_mutex_unlock(&js_devdata->runpool_mutex);
+ if (kctx_new != NULL)
+ {
+ osk_mutex_unlock( &js_kctx_new->ctx.jsctx_mutex );
+ }
+
+ /* Stop working on the target context, start working on the kctx_evict context */
+
+ osk_mutex_lock( &kctx_evict->jctx.sched_info.ctx.jsctx_mutex );
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+ release_result = kbasep_js_runpool_release_ctx_internal( kbdev, kctx_evict );
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+ /* Only requeue if actually descheduled, which is more robust in case
+ * something else retains it (e.g. two high priority contexts racing
+ * to evict the same lower priority context) */
+ if ( (release_result & KBASEP_JS_RELEASE_RESULT_WAS_DESCHEDULED) != 0u )
+ {
+ kbasep_js_runpool_requeue_or_kill_ctx( kbdev, kctx_evict );
+ }
+ osk_mutex_unlock( &kctx_evict->jctx.sched_info.ctx.jsctx_mutex );
+
+ /* release_result isn't propogated further:
+ * - the caller will be scheduling in a context anyway
+ * - which will also cause new jobs to run */
+
+ /* ctx fast start has taken place */
+ return;
+ }
+ }
+ }
+ osk_spinlock_irq_unlock(&js_devdata->runpool_irq.lock);
+ }
+
+ /* ctx fast start has not taken place */
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+ if (kctx_new != NULL)
+ {
+ osk_mutex_unlock( &js_kctx_new->ctx.jsctx_mutex );
+ }
+}
+
+mali_bool kbasep_js_add_job( kbase_context *kctx, kbase_jd_atom *atom )
+{
+ kbasep_js_kctx_info *js_kctx_info;
+ kbase_device *kbdev;
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_policy *js_policy;
+
+ mali_bool policy_queue_updated = MALI_FALSE;
+
+ OSK_ASSERT( kctx != NULL );
+ OSK_ASSERT( atom != NULL );
+
+ kbdev = kctx->kbdev;
+ js_devdata = &kbdev->js_data;
+ js_policy = &kbdev->js_data.policy;
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ osk_mutex_lock( &js_kctx_info->ctx.jsctx_mutex );
+ /* Policy-specific initialization of atoms (which cannot fail). Anything that
+ * could've failed must've been done at kbasep_jd_policy_init_job() time. */
+ kbasep_js_policy_register_job( js_policy, kctx, atom );
+
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+ {
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, JS_ADD_JOB, kctx, atom->user_atom, atom->jc,
+ kbasep_js_trace_get_refcnt(kbdev, kctx));
+ }
+
+ /* Refcount ctx.nr_jobs */
+ OSK_ASSERT( js_kctx_info->ctx.nr_jobs < U32_MAX );
+ ++(js_kctx_info->ctx.nr_jobs);
+
+ /* Setup any scheduling information */
+ kbasep_js_clear_job_retry_submit( atom );
+
+ /*
+ * Begin Runpool_irq transaction
+ */
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+ {
+ /* Context Attribute Refcounting */
+ kbasep_js_ctx_attr_ctx_retain_atom( kbdev, kctx, atom );
+
+ /* Enqueue the job in the policy, causing it to be scheduled if the
+ * parent context gets scheduled */
+ kbasep_js_policy_enqueue_job( js_policy, atom );
+ }
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+ /* End runpool_irq transaction */
+
+ if ( js_kctx_info->ctx.is_scheduled != MALI_FALSE )
+ {
+ /* Handle an already running context - try to run the new job, in case it
+ * matches requirements that aren't matched by any other job in the Run
+ * Pool */
+ kbasep_js_try_run_next_job( kbdev );
+ }
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+
+ if ( js_kctx_info->ctx.is_scheduled == MALI_FALSE && js_kctx_info->ctx.nr_jobs == 1 )
+ {
+ /* Handle Refcount going from 0 to 1: schedule the context on the Policy Queue */
+ OSK_ASSERT( js_kctx_info->ctx.is_scheduled == MALI_FALSE );
+
+ OSK_PRINT_INFO(OSK_BASE_JM, "JS: Enqueue Context %p", kctx );
+
+ /* This context is becoming active */
+ kbase_pm_context_active(kctx->kbdev);
+
+ osk_mutex_lock( &js_devdata->queue_mutex );
+ kbasep_js_policy_enqueue_ctx( js_policy, kctx );
+ osk_mutex_unlock( &js_devdata->queue_mutex );
+ /* If the runpool is full and this job has a higher priority than the non-running
+ * job in the runpool - evict it so this higher priority job starts faster */
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+
+ /* Fast-starting requires the jsctx_mutex to be dropped, because it works on multiple ctxs */
+ kbasep_js_runpool_attempt_fast_start_ctx( kbdev, kctx );
+
+ /* NOTE: Potentially, we can make the scheduling of the head context
+ * happen in a work-queue if we need to wait for the PM to power
+ * up. Also need logic to submit nothing until PM really has completed
+ * powering up. */
+
+ /* Policy Queue was updated - caller must try to schedule the head context */
+ policy_queue_updated = MALI_TRUE;
+ }
+ else
+ {
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+ }
+
+ return policy_queue_updated;
+}
+
+void kbasep_js_remove_job( kbase_context *kctx, kbase_jd_atom *atom )
+{
+ kbasep_js_kctx_info *js_kctx_info;
+ kbase_device *kbdev;
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_policy *js_policy;
+ mali_bool attr_state_changed;
+
+ OSK_ASSERT( kctx != NULL );
+ OSK_ASSERT( atom != NULL );
+
+ kbdev = kctx->kbdev;
+ js_devdata = &kbdev->js_data;
+ js_policy = &kbdev->js_data.policy;
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ {
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, JS_REMOVE_JOB, kctx, atom->user_atom, atom->jc,
+ kbasep_js_trace_get_refcnt(kbdev, kctx));
+ }
+
+ /* De-refcount ctx.nr_jobs */
+ OSK_ASSERT( js_kctx_info->ctx.nr_jobs > 0 );
+ --(js_kctx_info->ctx.nr_jobs);
+
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+ attr_state_changed = kbasep_js_ctx_attr_ctx_release_atom( kbdev, kctx, atom );
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+
+ /* De-register the job from the system */
+ kbasep_js_policy_deregister_job( js_policy, kctx, atom );
+
+ if ( attr_state_changed != MALI_FALSE )
+ {
+ /* A change in runpool ctx attributes might mean we can run more jobs
+ * than before. */
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+ kbasep_js_try_run_next_job( kbdev );
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+ }
+
+}
+
+
+mali_bool kbasep_js_runpool_retain_ctx( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ mali_bool result;
+ OSK_ASSERT( kbdev != NULL );
+ js_devdata = &kbdev->js_data;
+
+ /* KBASE_TRACE_ADD_REFCOUNT( kbdev, JS_RETAIN_CTX, kctx, NULL, 0,
+ kbasep_js_trace_get_refcnt(kbdev, kctx)); */
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+ result = kbasep_js_runpool_retain_ctx_nolock( kbdev, kctx );
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+
+ return result;
+}
+
+
+kbase_context* kbasep_js_runpool_lookup_ctx( kbase_device *kbdev, int as_nr )
+{
+ kbasep_js_device_data *js_devdata;
+ kbase_context *found_kctx = NULL;
+ kbasep_js_per_as_data *js_per_as_data;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( 0 <= as_nr && as_nr < BASE_MAX_NR_AS );
+ js_devdata = &kbdev->js_data;
+ js_per_as_data = &js_devdata->runpool_irq.per_as_data[as_nr];
+
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+
+ found_kctx = js_per_as_data->kctx;
+
+ if ( found_kctx != NULL )
+ {
+ ++(js_per_as_data->as_busy_refcount);
+ }
+
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+
+ return found_kctx;
+}
+
+/**
+ * Internal function to release the reference on a ctx, only taking the runpool and as transaction mutexes
+ *
+ * This does none of the followup actions for scheduling:
+ * - It does not schedule in a new context
+ * - It does not start more jobs running in the case of an ctx-attribute state change
+ * - It does not requeue or handling dying contexts
+ *
+ * For those tasks, just call kbasep_js_runpool_release_ctx() instead
+ *
+ * Requires:
+ * - Context is scheduled in, and kctx->as_nr matches kctx_as_nr
+ * - Context has a non-zero refcount
+ * - Caller holds js_kctx_info->ctx.jsctx_mutex
+ * - Caller holds js_devdata->runpool_mutex
+ */
+STATIC kbasep_js_release_result kbasep_js_runpool_release_ctx_internal( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_kctx_info *js_kctx_info;
+ kbasep_js_policy *js_policy;
+ kbasep_js_per_as_data *js_per_as_data;
+
+ kbasep_js_release_result release_result = 0u;
+ int kctx_as_nr;
+ kbase_as *current_as;
+ int new_ref_count;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+ js_kctx_info = &kctx->jctx.sched_info;
+ js_devdata = &kbdev->js_data;
+ js_policy = &kbdev->js_data.policy;
+
+ /* Ensure context really is scheduled in */
+ OSK_ASSERT( js_kctx_info->ctx.is_scheduled != MALI_FALSE );
+
+ /* kctx->as_nr and js_per_as_data are only read from here. The caller's
+ * js_ctx_mutex provides a barrier that ensures they are up-to-date.
+ *
+ * They will not change whilst we're reading them, because the refcount
+ * is non-zero (and we ASSERT on that last fact).
+ */
+ kctx_as_nr = kctx->as_nr;
+ OSK_ASSERT( kctx_as_nr != KBASEP_AS_NR_INVALID );
+ js_per_as_data = &js_devdata->runpool_irq.per_as_data[kctx_as_nr];
+ OSK_ASSERT( js_per_as_data->as_busy_refcount > 0 );
+
+ /*
+ * Transaction begins on AS and runpool_irq
+ *
+ * Assert about out calling contract
+ */
+ current_as = &kbdev->as[kctx_as_nr];
+ osk_mutex_lock( ¤t_as->transaction_mutex );
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+ OSK_ASSERT( kctx_as_nr == kctx->as_nr );
+ OSK_ASSERT( js_per_as_data->as_busy_refcount > 0 );
+
+ /* Update refcount */
+ new_ref_count = --(js_per_as_data->as_busy_refcount);
+
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, JS_RELEASE_CTX, kctx, NULL, 0u,
+ new_ref_count);
+
+ if ( new_ref_count == 1 && kctx->jctx.sched_info.ctx.flags & KBASE_CTX_FLAG_PRIVILEGED )
+ {
+ /* Context is kept scheduled into an address space even when there are no jobs, in this case we have
+ * to handle the situation where all jobs have been evicted from the GPU and submission is disabled.
+ *
+ * At this point we re-enable submission to allow further jobs to be executed
+ */
+ kbasep_js_set_submit_allowed( js_devdata, kctx );
+ }
+
+ /* Make a set of checks to see if the context should be scheduled out */
+ if ( new_ref_count == 0
+ && ( kctx->jctx.sched_info.ctx.nr_jobs == 0
+ || kbasep_js_is_submit_allowed( js_devdata, kctx ) == MALI_FALSE ) )
+ {
+ /* Last reference, and we've been told to remove this context from the Run Pool */
+ OSK_PRINT_INFO(OSK_BASE_JM, "JS: RunPool Remove Context %p because as_busy_refcount=%d, jobs=%d, allowed=%d",
+ kctx,
+ new_ref_count,
+ js_kctx_info->ctx.nr_jobs,
+ kbasep_js_is_submit_allowed( js_devdata, kctx ) );
+
+ kbasep_js_policy_runpool_remove_ctx( js_policy, kctx );
+
+ /* Stop any more refcounts occuring on the context */
+ js_per_as_data->kctx = NULL;
+
+ /* Ensure we prevent the context from submitting any new jobs
+ * e.g. from kbasep_js_try_run_next_job_on_slot_irq_nolock() */
+ kbasep_js_clear_submit_allowed( js_devdata, kctx );
+
+ /* Disable the MMU on the affected address space, and indicate it's invalid */
+ kbase_mmu_disable( kctx );
+
+#if MALI_GATOR_SUPPORT
+ kbase_trace_mali_mmu_as_released(kctx->as_nr);
+#endif
+
+ kctx->as_nr = KBASEP_AS_NR_INVALID;
+
+ /* Ctx Attribute handling */
+ if ( kbasep_js_ctx_attr_runpool_release_ctx( kbdev, kctx ) != MALI_FALSE )
+ {
+ release_result |= KBASEP_JS_RELEASE_RESULT_CTX_ATTR_CHANGE;
+ }
+
+ /*
+ * Transaction ends on AS and runpool_irq:
+ *
+ * By this point, the AS-related data is now clear and ready for re-use.
+ *
+ * Since releases only occur once for each previous successful retain, and no more
+ * retains are allowed on this context, no other thread will be operating in this
+ * code whilst we are
+ */
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+ osk_mutex_unlock( ¤t_as->transaction_mutex );
+
+ /* Free up the address space */
+ js_devdata->as_free |= ((u16)(1u << kctx_as_nr));
+ /* Note: Don't reuse kctx_as_nr now */
+
+ /* update book-keeping info */
+ runpool_dec_context_count( kbdev, kctx );
+ js_kctx_info->ctx.is_scheduled = MALI_FALSE;
+ /* Signal any waiter that the context is not scheduled, so is safe for
+ * termination - once the jsctx_mutex is also dropped, and jobs have
+ * finished. */
+ osk_waitq_set( &js_kctx_info->ctx.not_scheduled_waitq );
+ osk_waitq_clear( &js_kctx_info->ctx.scheduled_waitq );
+
+ /* Queue an action to occur after we've dropped the lock */
+ release_result |= KBASEP_JS_RELEASE_RESULT_WAS_DESCHEDULED;
+
+ }
+ else
+ {
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+ osk_mutex_unlock( ¤t_as->transaction_mutex );
+ }
+
+ return release_result;
+}
+
+void kbasep_js_runpool_requeue_or_kill_ctx( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_policy *js_policy;
+ kbasep_js_kctx_info *js_kctx_info;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+ js_kctx_info = &kctx->jctx.sched_info;
+ js_policy = &kbdev->js_data.policy;
+ js_devdata = &kbdev->js_data;
+
+ /* This is called if and only if you've you've detached the context from
+ * the Runpool or the Policy Queue, and not added it back to the Runpool */
+ OSK_ASSERT( js_kctx_info->ctx.is_scheduled == MALI_FALSE );
+
+ if ( js_kctx_info->ctx.is_dying != MALI_FALSE )
+ {
+ /* Dying: kill and idle the context */
+
+ /* Notify PM that a context has gone idle */
+ OSK_PRINT_INFO(OSK_BASE_JM, "JS: Idling Context %p (not requeued)", kctx );
+ kbase_pm_context_idle(kbdev);
+
+ /* The killing happens asynchronously */
+ OSK_PRINT_INFO(OSK_BASE_JM, "JS: ** Killing Context %p on RunPool Remove **", kctx );
+ kbasep_js_policy_kill_all_ctx_jobs( js_policy, kctx );
+ }
+ else if ( js_kctx_info->ctx.nr_jobs > 0 )
+ {
+ /* Not dying, has jobs: add back to the queue */
+ OSK_PRINT_INFO(OSK_BASE_JM, "JS: Requeue Context %p", kctx );
+ osk_mutex_lock( &js_devdata->queue_mutex );
+ kbasep_js_policy_enqueue_ctx( js_policy, kctx );
+ osk_mutex_unlock( &js_devdata->queue_mutex );
+ }
+ else
+ {
+ /* Not dying, no jobs: PM-idle the context, don't add back to the queue */
+ OSK_PRINT_INFO(OSK_BASE_JM, "JS: Idling Context %p (not requeued)", kctx );
+ kbase_pm_context_idle(kbdev);
+ }
+}
+
+void kbasep_js_runpool_release_ctx( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_kctx_info *js_kctx_info;
+
+ kbasep_js_release_result release_result;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+ js_kctx_info = &kctx->jctx.sched_info;
+ js_devdata = &kbdev->js_data;
+
+ osk_mutex_lock( &js_kctx_info->ctx.jsctx_mutex );
+
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+ release_result = kbasep_js_runpool_release_ctx_internal( kbdev, kctx );
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+
+ /* Do we have an action queued whilst the lock was held? */
+ if ( (release_result & KBASEP_JS_RELEASE_RESULT_WAS_DESCHEDULED) != 0u )
+ {
+ kbasep_js_runpool_requeue_or_kill_ctx( kbdev, kctx) ;
+ }
+ /* We've finished with this context for now, so drop the lock for it. */
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+
+ if ( (release_result & KBASEP_JS_RELEASE_RESULT_WAS_DESCHEDULED) != 0u )
+ {
+ /* We've freed up an address space, so let's try to schedule in another
+ * context
+ *
+ * Note: if there's a context to schedule in, then it also tries to run
+ * another job, in case the new context has jobs satisfying requirements
+ * that no other context/job in the runpool does */
+ kbasep_js_try_schedule_head_ctx( kbdev );
+ }
+
+ if ( (release_result & KBASEP_JS_RELEASE_RESULT_CTX_ATTR_CHANGE) != 0u )
+ {
+ /* A change in runpool ctx attributes might mean we can run more jobs
+ * than before - and this needs to be done when the above
+ * try_schedule_head_ctx() had no contexts to run */
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+ kbasep_js_try_run_next_job( kbdev );
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+ }
+
+}
+
+/**
+ * @brief Handle retaining cores for power management and affinity management,
+ * ensuring that cores are powered up and won't violate affinity restrictions.
+ *
+ * This function enters at the following @ref kbase_atom_coreref_state states:
+ *
+ * - NO_CORES_REQUESTED,
+ * - WAITING_FOR_REQUESTED_CORES,
+ * - RECHECK_AFFINITY,
+ *
+ * The transitions are as folows:
+ * - NO_CORES_REQUESTED -> WAITING_FOR_REQUESTED_CORES
+ * - WAITING_FOR_REQUESTED_CORES -> ( WAITING_FOR_REQUESTED_CORES or RECHECK_AFFINITY )
+ * - RECHECK_AFFINITY -> ( WAITING_FOR_REQUESTED_CORES or CHECK_AFFINITY_VIOLATIONS )
+ * - CHECK_AFFINITY_VIOLATIONS -> ( RECHECK_AFFINITY or READY )
+ *
+ * The caller must hold:
+ * - kbasep_js_device_data::runpool_irq::lock
+ *
+ * @return MALI_FALSE when the function makes a transition to the same or lower state, indicating
+ * that the cores are not ready.
+ * @return MALI_TRUE once READY state is reached, indicating that the cores are 'ready' and won't
+ * violate affinity restrictions.
+ *
+ */
+STATIC mali_bool kbasep_js_job_check_ref_cores(kbase_device *kbdev, int js, kbase_jd_atom *katom)
+{
+ u64 tiler_affinity = 0;
+ /* The most recently checked affinity. Having this at this scope allows us
+ * to guarantee that we've checked the affinity in this function call. */
+ u64 recently_chosen_affinity = 0;
+
+ if (katom->core_req & BASE_JD_REQ_T)
+ {
+ tiler_affinity = kbdev->tiler_present_bitmap;
+ }
+
+ /* NOTE: The following uses a number of FALLTHROUGHs to optimize the
+ * calls to this function. Ending of the function is indicated by BREAK OUT */
+ switch ( katom->coreref_state )
+ {
+ /* State when job is first attempted to be run */
+ case KBASE_ATOM_COREREF_STATE_NO_CORES_REQUESTED:
+ OSK_ASSERT( katom->affinity == 0 );
+ /* Compute affinity */
+ kbase_js_choose_affinity( &recently_chosen_affinity, kbdev, katom, js );
+
+ /* Request the cores */
+ if (MALI_ERROR_NONE != kbase_pm_request_cores( kbdev, recently_chosen_affinity, tiler_affinity ))
+ {
+ /* Failed to request cores, don't set the affinity so we try again and return */
+ KBASE_TRACE_ADD_SLOT_INFO( kbdev, JS_CORE_REF_REQUEST_CORES_FAILED, katom->kctx, katom->user_atom, katom->jc, js, (u32)recently_chosen_affinity );
+ /* *** BREAK OUT: No state transition *** */
+ break;
+ }
+
+ katom->affinity = recently_chosen_affinity;
+ /* Proceed to next state */
+ katom->coreref_state = KBASE_ATOM_COREREF_STATE_WAITING_FOR_REQUESTED_CORES;
+
+ /* ***FALLTHROUGH: TRANSITION TO HIGHER STATE*** */
+
+ case KBASE_ATOM_COREREF_STATE_WAITING_FOR_REQUESTED_CORES:
+ {
+ mali_bool cores_ready;
+ OSK_ASSERT( katom->affinity != 0 );
+
+ cores_ready = kbase_pm_register_inuse_cores( kbdev, katom->affinity, tiler_affinity );
+ if ( !cores_ready )
+ {
+ /* Stay in this state and return, to retry at this state later */
+ KBASE_TRACE_ADD_SLOT_INFO( kbdev, JS_CORE_REF_REGISTER_INUSE_FAILED, katom->kctx, katom->user_atom, katom->jc, js, (u32)katom->affinity );
+ /* *** BREAK OUT: No state transition *** */
+ break;
+ }
+ /* Proceed to next state */
+ katom->coreref_state = KBASE_ATOM_COREREF_STATE_RECHECK_AFFINITY;
+ }
+
+ /* ***FALLTHROUGH: TRANSITION TO HIGHER STATE*** */
+
+ case KBASE_ATOM_COREREF_STATE_RECHECK_AFFINITY:
+ OSK_ASSERT( katom->affinity != 0 );
+
+ /* Optimize out choosing the affinity twice in the same function call */
+ if ( recently_chosen_affinity == 0 )
+ {
+ /* See if the affinity changed since a previous call. */
+ kbase_js_choose_affinity( &recently_chosen_affinity, kbdev, katom, js );
+ }
+
+ /* Now see if this requires a different set of cores */
+ if ( recently_chosen_affinity != katom->affinity )
+ {
+ if (MALI_ERROR_NONE != kbase_pm_request_cores( kbdev, recently_chosen_affinity, tiler_affinity ))
+ {
+ /* Failed to request cores, rollback the previous gained set
+ * That also resets the state to NO_CORES_REQUESTED */
+ kbasep_js_job_check_deref_cores( kbdev, katom );
+ KBASE_TRACE_ADD_SLOT_INFO( kbdev, JS_CORE_REF_REQUEST_ON_RECHECK_FAILED, katom->kctx, katom->user_atom, katom->jc, js, (u32)recently_chosen_affinity );
+ /* *** BREAK OUT: Transition to lower state *** */
+ break;
+ }
+ else
+ {
+ mali_bool cores_ready;
+ /* Register new cores whislt we still hold the old ones, to minimize power transitions */
+ cores_ready = kbase_pm_register_inuse_cores( kbdev, recently_chosen_affinity, tiler_affinity );
+ kbasep_js_job_check_deref_cores( kbdev, katom );
+
+ /* Fixup the state that was reduced by deref_cores: */
+ katom->coreref_state = KBASE_ATOM_COREREF_STATE_RECHECK_AFFINITY;
+ katom->affinity = recently_chosen_affinity;
+ /* Now might be waiting for powerup again, with a new affinity */
+ if ( !cores_ready )
+ {
+ /* Return to previous state */
+ katom->coreref_state = KBASE_ATOM_COREREF_STATE_WAITING_FOR_REQUESTED_CORES;
+ KBASE_TRACE_ADD_SLOT_INFO( kbdev, JS_CORE_REF_REGISTER_ON_RECHECK_FAILED, katom->kctx, katom->user_atom, katom->jc, js, (u32)katom->affinity );
+ /* *** BREAK OUT: Transition to lower state *** */
+ break;
+ }
+ }
+ }
+ /* Proceed to next state */
+ katom->coreref_state = KBASE_ATOM_COREREF_STATE_CHECK_AFFINITY_VIOLATIONS;
+
+ /* ***FALLTHROUGH: TRANSITION TO HIGHER STATE*** */
+ case KBASE_ATOM_COREREF_STATE_CHECK_AFFINITY_VIOLATIONS:
+ OSK_ASSERT( katom->affinity != 0 );
+ OSK_ASSERT( katom->affinity == recently_chosen_affinity );
+
+ /* Note: this is where the caller must've taken the runpool_irq.lock */
+
+ /* Check for affinity violations - if there are any, then we just ask
+ * the caller to requeue and try again later */
+ if ( kbase_js_affinity_would_violate( kbdev, js, katom->affinity ) != MALI_FALSE )
+ {
+ /* Cause a re-attempt to submit from this slot on the next job complete */
+ kbase_js_affinity_slot_blocked_an_atom( kbdev, js );
+ /* Return to previous state */
+ katom->coreref_state = KBASE_ATOM_COREREF_STATE_RECHECK_AFFINITY;
+ /* *** BREAK OUT: Transition to lower state *** */
+ KBASE_TRACE_ADD_SLOT_INFO( kbdev, JS_CORE_REF_AFFINITY_WOULD_VIOLATE, katom->kctx, katom->user_atom, katom->jc, js, (u32)katom->affinity );
+ break;
+ }
+
+ /* No affinity violations would result, so the cores are ready */
+ katom->coreref_state = KBASE_ATOM_COREREF_STATE_READY;
+ /* *** BREAK OUT: Cores Ready *** */
+ break;
+
+ default:
+ OSK_ASSERT_MSG( MALI_FALSE, "Unhandled kbase_atom_coreref_state %d", katom->coreref_state );
+ break;
+ }
+
+ return (katom->coreref_state == KBASE_ATOM_COREREF_STATE_READY);
+}
+
+void kbasep_js_job_check_deref_cores(kbase_device *kbdev, struct kbase_jd_atom *katom)
+{
+ u64 tiler_affinity = 0;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( katom != NULL );
+
+ if (katom->core_req & BASE_JD_REQ_T)
+ {
+ tiler_affinity = kbdev->tiler_present_bitmap;
+ }
+
+ switch ( katom->coreref_state )
+ {
+ case KBASE_ATOM_COREREF_STATE_READY:
+ /* State where atom was submitted to the HW - just proceed to power-down */
+ OSK_ASSERT( katom->affinity != 0 );
+
+ /* *** FALLTHROUGH *** */
+
+ case KBASE_ATOM_COREREF_STATE_RECHECK_AFFINITY:
+ /* State where cores were registered */
+ OSK_ASSERT( katom->affinity != 0 );
+ kbase_pm_release_cores(kbdev, katom->affinity, tiler_affinity);
+
+ /* Note: We do not clear the state for kbase_js_affinity_slot_blocked_an_atom().
+ * That is handled after finishing the job. This might be slightly
+ * suboptimal for some corner cases, but is otherwise not a problem
+ * (and resolves itself after the next job completes). */
+
+ break;
+
+ case KBASE_ATOM_COREREF_STATE_WAITING_FOR_REQUESTED_CORES:
+ /* State where cores were requested, but not registered */
+ OSK_ASSERT( katom->affinity != 0 );
+ kbase_pm_unrequest_cores(kbdev, katom->affinity, tiler_affinity);
+ break;
+
+ case KBASE_ATOM_COREREF_STATE_NO_CORES_REQUESTED:
+ /* Initial state - nothing required */
+ OSK_ASSERT( katom->affinity == 0 );
+ break;
+
+ default:
+ OSK_ASSERT_MSG( MALI_FALSE, "Unhandled coreref_state: %d", katom->coreref_state );
+ break;
+ }
+
+ katom->affinity = 0;
+ katom->coreref_state = KBASE_ATOM_COREREF_STATE_NO_CORES_REQUESTED;
+}
+
+
+
+/*
+ * Note: this function is quite similar to kbasep_js_try_run_next_job_on_slot()
+ */
+mali_bool kbasep_js_try_run_next_job_on_slot_irq_nolock( kbase_device *kbdev, int js, s8 *submit_count )
+{
+ kbasep_js_device_data *js_devdata;
+ mali_bool tried_to_dequeue_jobs_but_failed = MALI_FALSE;
+ mali_bool cores_ready;
+
+ OSK_ASSERT( kbdev != NULL );
+
+ js_devdata = &kbdev->js_data;
+
+ /* The caller of this function may not be aware of Ctx Attribute state changes so we
+ * must recheck if the given slot is still valid. Otherwise do not try to run.
+ */
+ if (kbase_js_can_run_job_on_slot_no_lock( kbdev, js))
+ {
+ /* Keep submitting while there's space to run a job on this job-slot,
+ * and there are jobs to get that match its requirements (see 'break'
+ * statement below) */
+ while ( *submit_count < KBASE_JS_MAX_JOB_SUBMIT_PER_SLOT_PER_IRQ
+ && kbasep_jm_is_submit_slots_free( kbdev, js, NULL ) != MALI_FALSE )
+ {
+ kbase_jd_atom *dequeued_atom;
+ mali_bool has_job = MALI_FALSE;
+
+ /* Dequeue a job that matches the requirements */
+ has_job = kbasep_js_policy_dequeue_job_irq( kbdev, js, &dequeued_atom );
+
+ if ( has_job != MALI_FALSE )
+ {
+ /* NOTE: since the runpool_irq lock is currently held and acts across
+ * all address spaces, any context whose busy refcount has reached
+ * zero won't yet be scheduled out whilst we're trying to run jobs
+ * from it */
+ kbase_context *parent_ctx = dequeued_atom->kctx;
+ mali_bool retain_success;
+
+ /* Retain/power up the cores it needs, check if cores are ready */
+ cores_ready = kbasep_js_job_check_ref_cores( kbdev, js, dequeued_atom );
+
+ if ( cores_ready != MALI_TRUE )
+ {
+ /* The job can't be submitted until the cores are ready, requeue the job */
+ kbasep_js_policy_enqueue_job( &kbdev->js_data.policy, dequeued_atom );
+ break;
+ }
+
+ /* ASSERT that the Policy picked a job from an allowed context */
+ OSK_ASSERT( kbasep_js_is_submit_allowed( js_devdata, parent_ctx) );
+
+ /* Retain the context to stop it from being scheduled out
+ * This is released when the job finishes */
+ retain_success = kbasep_js_runpool_retain_ctx_nolock( kbdev, parent_ctx );
+ OSK_ASSERT( retain_success != MALI_FALSE );
+ CSTD_UNUSED( retain_success );
+
+ /* Retain the affinity on the slot */
+ kbase_js_affinity_retain_slot_cores( kbdev, js, dequeued_atom->affinity );
+
+ /* Check if this job needs the cycle counter enabled before submission */
+ kbasep_js_ref_permon_check_and_enable_cycle_counter( kbdev, dequeued_atom );
+
+ /* Submit the job */
+ kbase_job_submit_nolock( kbdev, dequeued_atom, js );
+
+ ++(*submit_count);
+ }
+ else
+ {
+ tried_to_dequeue_jobs_but_failed = MALI_TRUE;
+ /* No more jobs - stop submitting for this slot */
+ break;
+ }
+ }
+ }
+
+ /* Indicate whether a retry in submission should be tried on a different
+ * dequeue function. These are the reasons why it *must* happen:
+ *
+ * - kbasep_js_policy_dequeue_job_irq() couldn't get any jobs. In this case,
+ * kbasep_js_policy_dequeue_job() might be able to get jobs (must be done
+ * outside of IRQ)
+ * - kbasep_js_policy_dequeue_job_irq() got some jobs, but failed to get a
+ * job in the last call to it. Again, kbasep_js_policy_dequeue_job()
+ * might be able to get jobs.
+ * - the KBASE_JS_MAX_JOB_SUBMIT_PER_SLOT_PER_IRQ threshold was reached
+ * and new scheduling must be performed outside of IRQ mode.
+ *
+ * Failure to indicate this correctly could stop further jobs being processed.
+ *
+ * However, we do not _need_ to indicate a retry for the following:
+ * - kbasep_jm_is_submit_slots_free() was MALI_FALSE, indicating jobs were
+ * already running. When those jobs complete, that will still cause events
+ * that cause us to resume job submission.
+ * - kbase_js_can_run_job_on_slot_no_lock() was MALI_FALSE - this is for
+ * Ctx Attribute handling. That _can_ change outside of IRQ context, but
+ * is handled explicitly by kbasep_js_remove_job() and
+ * kbasep_js_runpool_release_ctx().
+ */
+ return (mali_bool)(tried_to_dequeue_jobs_but_failed || *submit_count >= KBASE_JS_MAX_JOB_SUBMIT_PER_SLOT_PER_IRQ);
+}
+
+void kbasep_js_try_run_next_job_on_slot( kbase_device *kbdev, int js )
+{
+ kbasep_js_device_data *js_devdata;
+ mali_bool has_job;
+ mali_bool cores_ready;
+
+ OSK_ASSERT( kbdev != NULL );
+
+ js_devdata = &kbdev->js_data;
+
+ if (js_devdata->nr_user_contexts_running == 0)
+ {
+ /* There are no contexts with jobs so return early */
+ return;
+ }
+
+ kbase_job_slot_lock(kbdev, js);
+
+ /* Keep submitting while there's space to run a job on this job-slot,
+ * and there are jobs to get that match its requirements (see 'break'
+ * statement below) */
+ if ( kbasep_jm_is_submit_slots_free( kbdev, js, NULL ) != MALI_FALSE )
+ {
+ /* Only lock the Run Pool whilst there's work worth doing */
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+
+ /* The caller of this function may not be aware of Ctx Attribute state changes so we
+ * must recheck if the given slot is still valid. Otherwise do not try to run.
+ */
+ if (kbase_js_can_run_job_on_slot_no_lock( kbdev, js))
+ {
+ do {
+ kbase_jd_atom *dequeued_atom;
+
+ /* Dequeue a job that matches the requirements */
+ has_job = kbasep_js_policy_dequeue_job( kbdev, js, &dequeued_atom );
+
+ if ( has_job != MALI_FALSE )
+ {
+ /* NOTE: since the runpool_irq lock is currently held and acts across
+ * all address spaces, any context whose busy refcount has reached
+ * zero won't yet be scheduled out whilst we're trying to run jobs
+ * from it */
+ kbase_context *parent_ctx = dequeued_atom->kctx;
+ mali_bool retain_success;
+
+ /* Retain/power up the cores it needs, check if cores are ready */
+ cores_ready = kbasep_js_job_check_ref_cores( kbdev, js, dequeued_atom );
+
+ if ( cores_ready != MALI_TRUE )
+ {
+ /* The job can't be submitted until the cores are ready, requeue the job */
+ kbasep_js_policy_enqueue_job( &kbdev->js_data.policy, dequeued_atom );
+ break;
+ }
+ /* ASSERT that the Policy picked a job from an allowed context */
+ OSK_ASSERT( kbasep_js_is_submit_allowed( js_devdata, parent_ctx) );
+
+ /* Retain the context to stop it from being scheduled out
+ * This is released when the job finishes */
+ retain_success = kbasep_js_runpool_retain_ctx_nolock( kbdev, parent_ctx );
+ OSK_ASSERT( retain_success != MALI_FALSE );
+ CSTD_UNUSED( retain_success );
+
+ /* Retain the affinity on the slot */
+ kbase_js_affinity_retain_slot_cores( kbdev, js, dequeued_atom->affinity );
+
+ /* Check if this job needs the cycle counter enabled before submission */
+ kbasep_js_ref_permon_check_and_enable_cycle_counter( kbdev, dequeued_atom );
+
+ /* Submit the job */
+ kbase_job_submit_nolock( kbdev, dequeued_atom, js );
+ }
+
+ } while ( kbasep_jm_is_submit_slots_free( kbdev, js, NULL ) != MALI_FALSE
+ && has_job != MALI_FALSE );
+ }
+
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+ }
+ kbase_job_slot_unlock(kbdev, js);
+}
+
+void kbasep_js_try_schedule_head_ctx( kbase_device *kbdev )
+{
+ kbasep_js_device_data *js_devdata;
+ mali_bool has_kctx;
+ kbase_context *head_kctx;
+ kbasep_js_kctx_info *js_kctx_info;
+ mali_bool is_runpool_full;
+
+ OSK_ASSERT( kbdev != NULL );
+
+ js_devdata = &kbdev->js_data;
+
+ /* Make a speculative check on the Run Pool - this MUST be repeated once
+ * we've obtained a context from the queue and reobtained the Run Pool
+ * lock */
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+ is_runpool_full = check_is_runpool_full(kbdev, NULL);
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+
+ if ( is_runpool_full != MALI_FALSE )
+ {
+ /* No free address spaces - nothing to do */
+ return;
+ }
+
+ /* Grab the context off head of queue - if there is one */
+ osk_mutex_lock( &js_devdata->queue_mutex );
+ has_kctx = kbasep_js_policy_dequeue_head_ctx( &js_devdata->policy, &head_kctx );
+ osk_mutex_unlock( &js_devdata->queue_mutex );
+
+ if ( has_kctx == MALI_FALSE )
+ {
+ /* No ctxs to run - nothing to do */
+ return;
+ }
+ js_kctx_info = &head_kctx->jctx.sched_info;
+
+ OSK_PRINT_INFO(OSK_BASE_JM, "JS: Dequeue Context %p", head_kctx );
+
+ /*
+ * Atomic transaction on the Context and Run Pool begins
+ */
+ osk_mutex_lock( &js_kctx_info->ctx.jsctx_mutex );
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+
+ /* Re-check to see if the Run Pool is full
+ * Not just to preserve atomicity, but to check against a specific context
+ * (some contexts are allowed in whereas others may not, due to HW workarounds) */
+ is_runpool_full = check_is_runpool_full(kbdev, head_kctx);
+ if ( is_runpool_full != MALI_FALSE )
+ {
+ /* No free address spaces - roll back the transaction so far and return */
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+
+ kbasep_js_runpool_requeue_or_kill_ctx( kbdev, head_kctx );
+
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+ return;
+ }
+
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, JS_TRY_SCHEDULE_HEAD_CTX, head_kctx, NULL, 0u,
+ kbasep_js_trace_get_refcnt(kbdev, head_kctx));
+
+
+ /* update book-keeping info */
+ js_kctx_info->ctx.is_scheduled = MALI_TRUE;
+
+#if MALI_CUSTOMER_RELEASE == 0
+ if ( js_devdata->nr_user_contexts_running == 0 )
+ {
+ /* Only when there are no other contexts submitting jobs:
+ * Latch in run-time job scheduler timeouts that were set through js_timeouts sysfs file */
+ if (kbdev->js_soft_stop_ticks != 0)
+ {
+ js_devdata->soft_stop_ticks = kbdev->js_soft_stop_ticks;
+ }
+ if (kbdev->js_hard_stop_ticks_ss != 0)
+ {
+ js_devdata->hard_stop_ticks_ss = kbdev->js_hard_stop_ticks_ss;
+ }
+ if (kbdev->js_hard_stop_ticks_nss != 0)
+ {
+ js_devdata->hard_stop_ticks_nss = kbdev->js_hard_stop_ticks_nss;
+ }
+ if (kbdev->js_reset_ticks_ss != 0)
+ {
+ js_devdata->gpu_reset_ticks_ss = kbdev->js_reset_ticks_ss;
+ }
+ if (kbdev->js_reset_ticks_nss != 0)
+ {
+ js_devdata->gpu_reset_ticks_nss = kbdev->js_reset_ticks_nss;
+ }
+ }
+#endif
+
+ runpool_inc_context_count( kbdev, head_kctx );
+ /* Cause any future waiter-on-termination to wait until the context is
+ * descheduled */
+ osk_waitq_clear( &js_kctx_info->ctx.not_scheduled_waitq );
+ osk_waitq_set( &js_kctx_info->ctx.scheduled_waitq );
+
+ /* Do all the necessaries to pick the address space (inc. update book-keeping info)
+ * Add the context to the Run Pool, and allow it to run jobs */
+ assign_and_activate_kctx_addr_space( kbdev, head_kctx );
+
+ if ( (js_kctx_info->ctx.flags & KBASE_CTX_FLAG_PRIVILEGED) != 0 )
+ {
+ /* We need to retain it to keep the corresponding address space */
+ kbasep_js_runpool_retain_ctx(kbdev, head_kctx);
+ }
+
+ /* Try to run the next job, in case this context has jobs that match the
+ * job slot requirements, but none of the other currently running contexts
+ * do */
+ kbasep_js_try_run_next_job( kbdev );
+
+ /* Transaction complete */
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+ /* Note: after this point, the context could potentially get scheduled out immediately */
+}
+
+void kbasep_js_schedule_privileged_ctx( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_kctx_info *js_kctx_info;
+ kbasep_js_device_data *js_devdata;
+ mali_bool is_scheduled;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ js_devdata = &kbdev->js_data;
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ osk_mutex_lock( &js_kctx_info->ctx.jsctx_mutex );
+ /* Mark the context as privileged */
+ js_kctx_info->ctx.flags |= KBASE_CTX_FLAG_PRIVILEGED;
+
+ is_scheduled = js_kctx_info->ctx.is_scheduled;
+ if ( is_scheduled == MALI_FALSE )
+ {
+ mali_bool is_runpool_full;
+
+ /* Add the context to the runpool */
+ osk_mutex_lock( &js_devdata->queue_mutex );
+ kbasep_js_policy_enqueue_ctx( &js_devdata->policy, kctx );
+ osk_mutex_unlock( &js_devdata->queue_mutex );
+
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+ {
+ is_runpool_full = check_is_runpool_full( kbdev, kctx);
+ if ( is_runpool_full != MALI_FALSE )
+ {
+ /* Evict jobs from the NEXT registers to free an AS asap */
+ kbasep_js_runpool_evict_next_jobs( kbdev );
+ }
+ }
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+ /* Fast-starting requires the jsctx_mutex to be dropped, because it works on multiple ctxs */
+
+ if ( is_runpool_full != MALI_FALSE )
+ {
+ /* Evict non-running contexts from the runpool */
+ kbasep_js_runpool_attempt_fast_start_ctx( kbdev, NULL );
+ }
+ /* Try to schedule the context in */
+ kbasep_js_try_schedule_head_ctx( kbdev );
+
+ /* Wait for the context to be scheduled in */
+ osk_waitq_wait(&kctx->jctx.sched_info.ctx.scheduled_waitq);
+ }
+ else
+ {
+ /* Already scheduled in - We need to retain it to keep the corresponding address space */
+ kbasep_js_runpool_retain_ctx(kbdev, kctx);
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+
+ }
+}
+
+void kbasep_js_release_privileged_ctx( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_kctx_info *js_kctx_info;
+ OSK_ASSERT( kctx != NULL );
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ /* We don't need to use the address space anymore */
+ osk_mutex_lock( &js_kctx_info->ctx.jsctx_mutex );
+ js_kctx_info->ctx.flags &= (~KBASE_CTX_FLAG_PRIVILEGED);
+ osk_mutex_unlock( &js_kctx_info->ctx.jsctx_mutex );
+
+ /* Release the context - it wil lbe scheduled out if there is no pending job */
+ kbasep_js_runpool_release_ctx(kbdev, kctx);
+}
+
+
+void kbasep_js_job_done_slot_irq( kbase_jd_atom *katom, int slot_nr, kbasep_js_tick *end_timestamp, mali_bool start_new_jobs )
+{
+ kbase_device *kbdev;
+ kbasep_js_policy *js_policy;
+ kbasep_js_device_data *js_devdata;
+ mali_bool submit_retry_needed = MALI_TRUE; /* If we don't start jobs here, start them from the workqueue */
+ kbasep_js_tick tick_diff;
+ u32 microseconds_spent = 0u;
+ kbase_context *parent_ctx;
+
+ OSK_ASSERT(katom);
+ parent_ctx = katom->kctx;
+ OSK_ASSERT(parent_ctx);
+ kbdev = parent_ctx->kbdev;
+ OSK_ASSERT(kbdev);
+
+ js_devdata = &kbdev->js_data;
+ js_policy = &kbdev->js_data.policy;
+
+ /*
+ * Release resources before submitting new jobs (bounds the refcount of
+ * the resource to BASE_JM_SUBMIT_SLOTS)
+ */
+#if MALI_GATOR_SUPPORT
+ kbase_trace_mali_job_slots_event(GATOR_MAKE_EVENT(GATOR_JOB_SLOT_STOP, slot_nr));
+#endif
+
+ if (katom->poking && kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8316))
+ {
+ OSK_ASSERT(parent_ctx->as_nr != KBASEP_AS_NR_INVALID);
+ kbase_as_poking_timer_release(&kbdev->as[parent_ctx->as_nr]);
+ katom->poking = 0;
+ }
+
+ /* Check if submitted jobs no longer require the cycle counter to be enabled */
+ kbasep_js_deref_permon_check_and_disable_cycle_counter( kbdev, katom );
+
+ /* Release the affinity from the slot - must happen before next submission to this slot */
+ kbase_js_affinity_release_slot_cores( kbdev, slot_nr, katom->affinity );
+ kbase_js_debug_log_current_affinities( kbdev );
+ /* Calculate the job's time used */
+ if ( end_timestamp != NULL )
+ {
+ /* Only calculating it for jobs that really run on the HW (e.g. removed
+ * from next jobs never actually ran, so really did take zero time) */
+ tick_diff = *end_timestamp - katom->start_timestamp;
+ microseconds_spent = kbasep_js_convert_js_ticks_to_us( tick_diff );
+ /* Round up time spent to the minimum timer resolution */
+ if (microseconds_spent < KBASEP_JS_TICK_RESOLUTION_US)
+ {
+ microseconds_spent = KBASEP_JS_TICK_RESOLUTION_US;
+ }
+ }
+
+ /* Lock the runpool_irq for modifying the runpool_irq data */
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+
+ /* Log the result of the job (completion status, and time spent). */
+ kbasep_js_policy_log_job_result( js_policy, katom, microseconds_spent );
+ /* Determine whether the parent context's timeslice is up */
+ if ( kbasep_js_policy_should_remove_ctx( js_policy, parent_ctx ) != MALI_FALSE )
+ {
+ kbasep_js_clear_submit_allowed( js_devdata, parent_ctx );
+ }
+
+ if ( start_new_jobs != MALI_FALSE )
+ {
+ /* Submit a new job (if there is one) to help keep the GPU's HEAD and NEXT registers full */
+ KBASE_TRACE_ADD_SLOT( kbdev, JS_JOB_DONE_TRY_RUN_NEXT_JOB, parent_ctx, katom->user_atom, katom->jc, slot_nr);
+
+ submit_retry_needed = kbasep_js_try_run_next_job_on_slot_irq_nolock(
+ kbdev,
+ slot_nr,
+ &kbdev->slot_submit_count_irq[slot_nr] );
+ }
+
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+ /* We've finished modifying runpool_irq data, so the lock is dropped */
+
+ if ( submit_retry_needed != MALI_FALSE || katom->event.event_code == BASE_JD_EVENT_STOPPED )
+ {
+ /* The extra condition on STOPPED jobs is needed because they may be
+ * the only job present, but they won't get re-run until the JD work
+ * queue activates. Crucially, work queues can run items out of order
+ * e.g. on different CPUs, so being able to submit from the IRQ handler
+ * is not a good indication that we don't need to run jobss; the
+ * submitted job could be processed on the work-queue *before* the
+ * stopped job, even though it was submitted after.
+ *
+ * Therefore, we must try to run it, otherwise it might not get run at
+ * all after this. */
+
+ KBASE_TRACE_ADD_SLOT( kbdev, JS_JOB_DONE_RETRY_NEEDED, parent_ctx, katom->user_atom, katom->jc, slot_nr);
+ kbasep_js_set_job_retry_submit_slot( katom, slot_nr );
+ }
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_js.h
+ * Job Scheduler APIs.
+ */
+
+#ifndef _KBASE_JS_H_
+#define _KBASE_JS_H_
+
+#include <malisw/mali_malisw.h>
+#include <osk/mali_osk.h>
+
+#include "mali_kbase_js_defs.h"
+#include "mali_kbase_js_policy.h"
+#include "mali_kbase_defs.h"
+
+#include "mali_kbase_js_ctx_attr.h"
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_kbase_api
+ * @{
+ */
+
+/**
+ * @addtogroup kbase_js Job Scheduler Internal APIs
+ * @{
+ *
+ * These APIs are Internal to KBase and are available for use by the
+ * @ref kbase_js_policy "Job Scheduler Policy APIs"
+ */
+
+/**
+ * @brief Initialize the Job Scheduler
+ *
+ * The kbasep_js_device_data sub-structure of \a kbdev must be zero
+ * initialized before passing to the kbasep_js_devdata_init() function. This is
+ * to give efficient error path code.
+ */
+mali_error kbasep_js_devdata_init( kbase_device *kbdev );
+
+/**
+ * @brief Halt the Job Scheduler.
+ *
+ * It is safe to call this on \a kbdev even if it the kbasep_js_device_data
+ * sub-structure was never initialized/failed initialization, to give efficient
+ * error-path code.
+ *
+ * For this to work, the kbasep_js_device_data sub-structure of \a kbdev must
+ * be zero initialized before passing to the kbasep_js_devdata_init()
+ * function. This is to give efficient error path code.
+ *
+ * It is a Programming Error to call this whilst there are still kbase_context
+ * structures registered with this scheduler.
+ *
+ */
+void kbasep_js_devdata_halt( kbase_device * kbdev);
+
+/**
+ * @brief Terminate the Job Scheduler
+ *
+ * It is safe to call this on \a kbdev even if it the kbasep_js_device_data
+ * sub-structure was never initialized/failed initialization, to give efficient
+ * error-path code.
+ *
+ * For this to work, the kbasep_js_device_data sub-structure of \a kbdev must
+ * be zero initialized before passing to the kbasep_js_devdata_init()
+ * function. This is to give efficient error path code.
+ *
+ * It is a Programming Error to call this whilst there are still kbase_context
+ * structures registered with this scheduler.
+ */
+void kbasep_js_devdata_term( kbase_device *kbdev );
+
+
+/**
+ * @brief Initialize the Scheduling Component of a kbase_context on the Job Scheduler.
+ *
+ * This effectively registers a kbase_context with a Job Scheduler.
+ *
+ * It does not register any jobs owned by the kbase_context with the scheduler.
+ * Those must be separately registered by kbasep_js_add_job().
+ *
+ * The kbase_context must be zero intitialized before passing to the
+ * kbase_js_init() function. This is to give efficient error path code.
+ */
+mali_error kbasep_js_kctx_init( kbase_context *kctx );
+
+/**
+ * @brief Terminate the Scheduling Component of a kbase_context on the Job Scheduler
+ *
+ * This effectively de-registers a kbase_context from its Job Scheduler
+ *
+ * It is safe to call this on a kbase_context that has never had or failed
+ * initialization of its jctx.sched_info member, to give efficient error-path
+ * code.
+ *
+ * For this to work, the kbase_context must be zero intitialized before passing
+ * to the kbase_js_init() function.
+ *
+ * It is a Programming Error to call this whilst there are still jobs
+ * registered with this context.
+ */
+void kbasep_js_kctx_term( kbase_context *kctx );
+
+/**
+ * @brief Add a job chain to the Job Scheduler, and take necessary actions to
+ * schedule the context/run the job.
+ *
+ * This atomically does the following:
+ * - Update the numbers of jobs information (including NSS state changes)
+ * - Add the job to the run pool if necessary (part of init_job)
+ *
+ * Once this is done, then an appropriate action is taken:
+ * - If the ctx is scheduled, it attempts to start the next job (which might be
+ * this added job)
+ * - Otherwise, and if this is the first job on the context, it enqueues it on
+ * the Policy Queue
+ *
+ * The Policy's Queue can be updated by this in the following ways:
+ * - In the above case that this is the first job on the context
+ * - If the job is high priority and the context is not scheduled, then it
+ * could cause the Policy to schedule out a low-priority context, allowing
+ * this context to be scheduled in.
+ *
+ * If the context is already scheduled on the RunPool, then adding a job to it
+ * is guarenteed not to update the Policy Queue. And so, the caller is
+ * guarenteed to not need to try scheduling a context from the Run Pool - it
+ * can safely assert that the result is MALI_FALSE.
+ *
+ * It is a programming error to have more than U32_MAX jobs in flight at a time.
+ *
+ * The following locking conditions are made on the caller:
+ * - it must \em not hold kbasep_js_kctx_info::ctx::jsctx_mutex.
+ * - it must \em not hold kbasep_js_device_data::runpool_irq::lock (as this will be
+ * obtained internally)
+ * - it must \em not hold kbasep_js_device_data::runpool_mutex (as this will be
+ * obtained internally)
+ * - it must \em not hold kbasep_jd_device_data::queue_mutex (again, it's used internally).
+ *
+ * @return MALI_TRUE indicates that the Policy Queue was updated, and so the
+ * caller will need to try scheduling a context onto the Run Pool.
+ * @return MALI_FALSE indicates that no updates were made to the Policy Queue,
+ * so no further action is required from the caller. This is \b always returned
+ * when the context is currently scheduled.
+ */
+mali_bool kbasep_js_add_job( kbase_context *kctx, kbase_jd_atom *atom );
+
+/**
+ * @brief Remove a job chain from the Job Scheduler
+ *
+ * Removing a job from the Scheduler can cause an NSS/SS state transition. In
+ * this case, slots that previously could not have jobs submitted to might now
+ * be submittable to. For this reason, and NSS/SS state transition will cause
+ * the Scheduler to try to submit new jobs on the jm_slots.
+ *
+ * It is a programming error to call this when:
+ * - \a atom is not a job belonging to kctx.
+ * - \a atom has already been removed from the Job Scheduler.
+ * - \a atom is still in the runpool:
+ * - it has not been killed with kbasep_js_policy_kill_all_ctx_jobs()
+ * - or, it has not been removed with kbasep_js_policy_dequeue_job()
+ * - or, it has not been removed with kbasep_js_policy_dequeue_job_irq()
+ *
+ * The following locking conditions are made on the caller:
+ * - it must hold kbasep_js_kctx_info::ctx::jsctx_mutex.
+ * - it must \em not hold kbasep_js_device_data::runpool_irq::lock, (as this will be
+ * obtained internally)
+ * - it must \em not hold kbasep_js_device_data::runpool_mutex (as this could be
+ obtained internally)
+ * - it must \em not hold kbdev->jm_slots[ \a js ].lock (as this could be
+ * obtained internally)
+ *
+ */
+void kbasep_js_remove_job( kbase_context *kctx, kbase_jd_atom *atom );
+
+/**
+ * @brief Refcount a context as being busy, preventing it from being scheduled
+ * out.
+ *
+ * @note This function can safely be called from IRQ context.
+ *
+ * The following locking conditions are made on the caller:
+ * - it must \em not hold the kbasep_js_device_data::runpool_irq::lock, because
+ * it will be used internally.
+ *
+ * @return value != MALI_FALSE if the retain succeeded, and the context will not be scheduled out.
+ * @return MALI_FALSE if the retain failed (because the context is being/has been scheduled out).
+ */
+mali_bool kbasep_js_runpool_retain_ctx( kbase_device *kbdev, kbase_context *kctx );
+
+/**
+ * @brief Refcount a context as being busy, preventing it from being scheduled
+ * out.
+ *
+ * @note This function can safely be called from IRQ context.
+ *
+ * The following locks must be held by the caller:
+ * - kbasep_js_device_data::runpool_irq::lock
+ *
+ * @return value != MALI_FALSE if the retain succeeded, and the context will not be scheduled out.
+ * @return MALI_FALSE if the retain failed (because the context is being/has been scheduled out).
+ */
+mali_bool kbasep_js_runpool_retain_ctx_nolock( kbase_device *kbdev, kbase_context *kctx );
+
+/**
+ * @brief Lookup a context in the Run Pool based upon its current address space
+ * and ensure that is stays scheduled in.
+ *
+ * The context is refcounted as being busy to prevent it from scheduling
+ * out. It must be released with kbasep_js_runpool_release_ctx() when it is no
+ * longer required to stay scheduled in.
+ *
+ * @note This function can safely be called from IRQ context.
+ *
+ * The following locking conditions are made on the caller:
+ * - it must \em not hold the kbasep_js_device_data::runpoool_irq::lock, because
+ * it will be used internally.
+ *
+ * @return a valid kbase_context on success, which has been refcounted as being busy.
+ * @return NULL on failure, indicating that no context was found in \a as_nr
+ */
+kbase_context* kbasep_js_runpool_lookup_ctx( kbase_device *kbdev, int as_nr );
+
+/**
+ * @brief Handling the requeuing/killing of a context that was evicted from the
+ * policy queue or runpool.
+ *
+ * This should be used whenever handing off a context that has been evicted
+ * from the policy queue or the runpool:
+ * - If the context is not dying and has jobs, it gets re-added to the policy
+ * queue
+ * - Otherwise, it is not added (but PM is informed that it is idle)
+ *
+ * In addition, if the context is dying the jobs are killed asynchronously.
+ *
+ * The following locking conditions are made on the caller:
+ * - it must hold kbasep_js_kctx_info::ctx::jsctx_mutex.
+ * - it must \em not hold kbasep_jd_device_data::queue_mutex (as this will be
+ * obtained internally)
+ */
+void kbasep_js_runpool_requeue_or_kill_ctx( kbase_device *kbdev, kbase_context *kctx );
+
+/**
+ * @brief Release a refcount of a context being busy, allowing it to be
+ * scheduled out.
+ *
+ * When the refcount reaches zero and the context \em might be scheduled out
+ * (depending on whether the Scheudling Policy has deemed it so, or if it has run
+ * out of jobs).
+ *
+ * If the context does get scheduled out, then The following actions will be
+ * taken as part of deschduling a context:
+ * - For the context being descheduled:
+ * - If the context is in the processing of dying (all the jobs are being
+ * removed from it), then descheduling also kills off any jobs remaining in the
+ * context.
+ * - If the context is not dying, and any jobs remain after descheduling the
+ * context then it is re-enqueued to the Policy's Queue.
+ * - Otherwise, the context is still known to the scheduler, but remains absent
+ * from the Policy Queue until a job is next added to it.
+ * - Once the context is descheduled, this also handles scheduling in a new
+ * context (if one is available), and if necessary, running a job from that new
+ * context.
+ *
+ * Unlike retaining a context in the runpool, this function \b cannot be called
+ * from IRQ context.
+ *
+ * It is a programming error to call this on a \a kctx that is not currently
+ * scheduled, or that already has a zero refcount.
+ *
+ * The following locking conditions are made on the caller:
+ * - it must \em not hold the kbasep_js_device_data::runpool_irq::lock, because
+ * it will be used internally.
+ * - it must \em not hold kbasep_js_kctx_info::ctx::jsctx_mutex.
+ * - it must \em not hold kbasep_js_device_data::runpool_mutex (as this will be
+ * obtained internally)
+ * - it must \em not hold the kbase_device::as[n].transaction_mutex (as this will be obtained internally)
+ * - it must \em not hold kbasep_jd_device_data::queue_mutex (as this will be
+ * obtained internally)
+ *
+ */
+void kbasep_js_runpool_release_ctx( kbase_device *kbdev, kbase_context *kctx );
+
+/**
+ * @brief Try to submit the next job on a \b particular slot whilst in IRQ
+ * context, and whilst the caller already holds the job-slot IRQ spinlock.
+ *
+ * \a *submit_count will be checked against
+ * KBASE_JS_MAX_JOB_SUBMIT_PER_SLOT_PER_IRQ to see whether too many jobs have
+ * been submitted. This is to prevent the IRQ handler looping over lots of GPU
+ * NULL jobs, which may complete whilst the IRQ handler is still processing. \a
+ * submit_count itself should point to kbase_device::slot_submit_count_irq[ \a js ],
+ * which is initialized to zero on entry to the IRQ handler.
+ *
+ * The following locks must be held by the caller:
+ * - kbasep_js_device_data::runpool_irq::lock
+ * - kbdev->jm_slots[ \a js ].lock
+ *
+ * @return truthful (i.e. != MALI_FALSE) if there was space to submit in the
+ * GPU, but we couldn't get a job from the Run Pool. This may be because the
+ * Run Pool needs maintenence outside of IRQ context. Therefore, this indicates
+ * that submission should be retried from a work-queue, by using
+ * kbasep_js_try_run_next_job_on_slot().
+ * @return MALI_FALSE if submission had no problems: the GPU is either already
+ * full of jobs in the HEAD and NEXT registers, or we were able to get enough
+ * jobs from the Run Pool to fill the GPU's HEAD and NEXT registers.
+ */
+mali_bool kbasep_js_try_run_next_job_on_slot_irq_nolock( kbase_device *kbdev, int js, s8 *submit_count );
+
+/**
+ * @brief Try to submit the next job on a particular slot, outside of IRQ context
+ *
+ * This obtains the Job Slot lock for the duration of the call only.
+ *
+ * Unlike kbasep_js_try_run_next_job_on_slot_irq_nolock(), there is no limit on
+ * submission, because eventually IRQ_THROTTLE will kick in to prevent us
+ * getting stuck in a loop of submitting GPU NULL jobs. This is because the IRQ
+ * handler will be delayed, and so this function will eventually fill up the
+ * space in our software 'submitted' slot (kbase_jm_slot::submitted).
+ *
+ * In addition, there's no return value - we'll run the maintenence functions
+ * on the Policy's Run Pool, but if there's nothing there after that, then the
+ * Run Pool is truely empty, and so no more action need be taken.
+ *
+ * The following locking conditions are made on the caller:
+ * - it must hold kbasep_js_device_data::runpool_mutex
+ * - it must \em not hold kbasep_js_device_data::runpool_irq::lock (as this will be
+ * obtained internally)
+ * - it must \em not hold kbdev->jm_slots[ \a js ].lock (as this will be
+ * obtained internally)
+ *
+ * @note The caller \em might be holding one of the
+ * kbasep_js_kctx_info::ctx::jsctx_mutex locks.
+ *
+ */
+void kbasep_js_try_run_next_job_on_slot( kbase_device *kbdev, int js );
+
+/**
+ * @brief Try to submit the next job for each slot in the system, outside of IRQ context
+ *
+ * This will internally call kbasep_js_try_run_next_job_on_slot(), so the same
+ * locking conditions on the caller are required.
+ *
+ * The following locking conditions are made on the caller:
+ * - it must hold kbasep_js_device_data::runpool_mutex
+ * - it must \em not hold kbasep_js_device_data::runpool_irq::lock (as this will be
+ * obtained internally)
+ * - it must \em not hold kbdev->jm_slots[ \a js ].lock (as this will be
+ * obtained internally)
+ *
+ * @note The caller \em might be holding one of the
+ * kbasep_js_kctx_info::ctx::jsctx_mutex locks.
+ *
+ */
+void kbasep_js_try_run_next_job( kbase_device *kbdev );
+
+/**
+ * @brief Try to schedule the next context onto the Run Pool
+ *
+ * This checks whether there's space in the Run Pool to accommodate a new
+ * context. If so, it attempts to dequeue a context from the Policy Queue, and
+ * submit this to the Run Pool.
+ *
+ * If the scheduling succeeds, then it also makes a call to
+ * kbasep_js_try_run_next_job(), in case the new context has jobs matching the
+ * job slot requirements, but no other currently scheduled context has such
+ * jobs.
+ *
+ * If any of these actions fail (Run Pool Full, Policy Queue empty, etc) then
+ * the function just returns normally.
+ *
+ * The following locking conditions are made on the caller:
+ * - it must \em not hold the kbasep_js_device_data::runpool_irq::lock, because
+ * it will be used internally.
+ * - it must \em not hold kbasep_js_device_data::runpool_mutex (as this will be
+ * obtained internally)
+ * - it must \em not hold the kbase_device::as[n].transaction_mutex (as this will be obtained internally)
+ * - it must \em not hold kbasep_jd_device_data::queue_mutex (again, it's used internally).
+ * - it must \em not hold kbasep_js_kctx_info::ctx::jsctx_mutex, because it will
+ * be used internally.
+ * - it must \em not hold kbdev->jm_slots[ \a js ].lock (as this will be
+ * obtained internally)
+ *
+ */
+void kbasep_js_try_schedule_head_ctx( kbase_device *kbdev );
+
+/**
+ * @brief Schedule in a privileged context
+ *
+ * This schedules a context in regardless of the context priority.
+ * If the runpool is full, a context will be forced out of the runpool and the function will wait
+ * for the new context to be scheduled in.
+ * The context will be kept scheduled in (and the corresponding address space reserved) until
+ * kbasep_js_release_privileged_ctx is called).
+ *
+ * The following locking conditions are made on the caller:
+ * - it must \em not hold the kbasep_js_device_data::runpool_irq::lock, because
+ * it will be used internally.
+ * - it must \em not hold kbasep_js_device_data::runpool_mutex (as this will be
+ * obtained internally)
+ * - it must \em not hold the kbase_device::as[n].transaction_mutex (as this will be obtained internally)
+ * - it must \em not hold kbasep_jd_device_data::queue_mutex (again, it's used internally).
+ * - it must \em not hold kbasep_js_kctx_info::ctx::jsctx_mutex, because it will
+ * be used internally.
+ * - it must \em not hold kbdev->jm_slots[ \a js ].lock (as this will be
+ * obtained internally)
+ *
+ */
+void kbasep_js_schedule_privileged_ctx( kbase_device *kbdev, kbase_context *kctx );
+
+/**
+ * @brief Release a privileged context, allowing it to be scheduled out.
+ *
+ * See kbasep_js_runpool_release_ctx for potential side effects.
+ *
+ * The following locking conditions are made on the caller:
+ * - it must \em not hold the kbasep_js_device_data::runpool_irq::lock, because
+ * it will be used internally.
+ * - it must \em not hold kbasep_js_kctx_info::ctx::jsctx_mutex.
+ * - it must \em not hold kbasep_js_device_data::runpool_mutex (as this will be
+ * obtained internally)
+ * - it must \em not hold the kbase_device::as[n].transaction_mutex (as this will be obtained internally)
+ *
+ */
+void kbasep_js_release_privileged_ctx( kbase_device *kbdev, kbase_context *kctx );
+
+/**
+ * @brief Handle the Job Scheduler component for the IRQ of a job finishing
+ *
+ * This does the following:
+ * -# Releases resources held by the atom
+ * -# if \a end_timestamp != NULL, updates the runpool's notion of time spent by a running ctx
+ * -# determines whether a context should be marked for scheduling out
+ * -# if start_new_jobs is true, tries to submit the next job on the slot
+ * (picking from all ctxs in the runpool)
+ *
+ * In addition, if submission didn't happen (the submit-from-IRQ function
+ * failed or start_new_jobs == MALI_FALSE), then this sets a message on katom
+ * that submission needs to be retried from the worker thread.
+ *
+ * Normally, the time calculated from end_timestamp is rounded up to the
+ * minimum time precision. Therefore, to ensure the job is recorded as not
+ * spending any time, then set end_timestamp to NULL. For example, this is necessary when
+ * evicting jobs from JSn_HEAD_NEXT (because they didn't actually run).
+ *
+ * NOTE: It's possible to move the steps (2) and (3) (inc calculating job's time
+ * used) into the worker (outside of IRQ context), but this may allow a context
+ * to use up to twice as much timeslice as is allowed by the policy. For
+ * policies that order by time spent, this is not a problem for overall
+ * 'fairness', but can still increase latency between contexts.
+ *
+ * The following locking conditions are made on the caller:
+ * - it must \em not hold the kbasep_js_device_data::runpoool_irq::lock, because
+ * it will be used internally.
+ * - it must hold kbdev->jm_slots[ \a slot_nr ].lock
+ */
+void kbasep_js_job_done_slot_irq( kbase_jd_atom *katom, int slot_nr, kbasep_js_tick *end_timestamp, mali_bool start_new_jobs );
+
+/**
+ * @brief Try to submit the next job on each slot
+ *
+ * The following locks may be used:
+ * - kbasep_js_device_data::runpool_mutex
+ * - kbasep_js_device_data::runpool_irq::lock
+ * - bdev->jm_slots[ \a js ].lock
+ */
+void kbase_js_try_run_jobs( kbase_device *kbdev );
+
+/**
+ * @brief Handle releasing cores for power management and affinity management,
+ * ensuring that cores are powered down and affinity tracking is updated.
+ *
+ * This must only be called on an atom that is not currently running, and has
+ * not been re-queued onto the context (and so does not need locking)
+ *
+ * This function enters at the following @ref kbase_atom_coreref_state states:
+ * - NO_CORES_REQUESTED
+ * - WAITING_FOR_REQUESTED_CORES
+ * - RECHECK_AFFINITY
+ * - READY
+ *
+ * It transitions the above states back to NO_CORES_REQUESTED by the end of the
+ * function call (possibly via intermediate states).
+ *
+ * No locks need be held by the caller, since this takes the necessary Power
+ * Management locks itself. The runpool_irq.lock is not taken (the work that
+ * requires it is handled by kbase_js_affinity_submit_to_blocked_slots() ).
+ *
+ * @note The corresponding kbasep_js_job_check_ref_cores() is private to the
+ * Job Scheduler, and is called automatically when running the next job.
+ */
+void kbasep_js_job_check_deref_cores(kbase_device *kbdev, struct kbase_jd_atom *katom);
+
+/*
+ * Helpers follow
+ */
+
+/**
+ * @brief Check that a context is allowed to submit jobs on this policy
+ *
+ * The purpose of this abstraction is to hide the underlying data size, and wrap up
+ * the long repeated line of code.
+ *
+ * As with any mali_bool, never test the return value with MALI_TRUE.
+ *
+ * The caller must hold kbasep_js_device_data::runpool_irq::lock.
+ */
+static INLINE mali_bool kbasep_js_is_submit_allowed( kbasep_js_device_data *js_devdata, kbase_context *kctx )
+{
+ u16 test_bit;
+
+ /* Ensure context really is scheduled in */
+ OSK_ASSERT( kctx->as_nr != KBASEP_AS_NR_INVALID );
+ OSK_ASSERT( kctx->jctx.sched_info.ctx.is_scheduled != MALI_FALSE );
+
+ test_bit = (u16)(1u << kctx->as_nr);
+
+ return (mali_bool)(js_devdata->runpool_irq.submit_allowed & test_bit);
+}
+
+/**
+ * @brief Allow a context to submit jobs on this policy
+ *
+ * The purpose of this abstraction is to hide the underlying data size, and wrap up
+ * the long repeated line of code.
+ *
+ * The caller must hold kbasep_js_device_data::runpool_irq::lock.
+ */
+static INLINE void kbasep_js_set_submit_allowed( kbasep_js_device_data *js_devdata, kbase_context *kctx )
+{
+ u16 set_bit;
+
+ /* Ensure context really is scheduled in */
+ OSK_ASSERT( kctx->as_nr != KBASEP_AS_NR_INVALID );
+ OSK_ASSERT( kctx->jctx.sched_info.ctx.is_scheduled != MALI_FALSE );
+
+ set_bit = (u16)(1u << kctx->as_nr);
+
+ OSK_PRINT_INFO(OSK_BASE_JM, "JS: Setting Submit Allowed on %p (as=%d)", kctx, kctx->as_nr );
+
+ js_devdata->runpool_irq.submit_allowed |= set_bit;
+}
+
+/**
+ * @brief Prevent a context from submitting more jobs on this policy
+ *
+ * The purpose of this abstraction is to hide the underlying data size, and wrap up
+ * the long repeated line of code.
+ *
+ * The caller must hold kbasep_js_device_data::runpool_irq::lock.
+ */
+static INLINE void kbasep_js_clear_submit_allowed( kbasep_js_device_data *js_devdata, kbase_context *kctx )
+{
+ u16 clear_bit;
+ u16 clear_mask;
+
+ /* Ensure context really is scheduled in */
+ OSK_ASSERT( kctx->as_nr != KBASEP_AS_NR_INVALID );
+ OSK_ASSERT( kctx->jctx.sched_info.ctx.is_scheduled != MALI_FALSE );
+
+ clear_bit = (u16)(1u << kctx->as_nr);
+ clear_mask = ~clear_bit;
+
+ OSK_PRINT_INFO(OSK_BASE_JM, "JS: Clearing Submit Allowed on %p (as=%d)", kctx, kctx->as_nr );
+
+ js_devdata->runpool_irq.submit_allowed &= clear_mask;
+}
+
+/**
+ * @brief Manage the 'retry_submit_on_slot' part of a kbase_jd_atom
+ */
+static INLINE void kbasep_js_clear_job_retry_submit( kbase_jd_atom *atom )
+{
+ atom->retry_submit_on_slot = -1;
+}
+
+static INLINE mali_bool kbasep_js_get_job_retry_submit_slot( kbase_jd_atom *atom, int *res )
+{
+ int js = atom->retry_submit_on_slot;
+ *res = js;
+ return (mali_bool)( js >= 0 );
+}
+
+static INLINE void kbasep_js_set_job_retry_submit_slot( kbase_jd_atom *atom, int js )
+{
+ OSK_ASSERT( 0 <= js && js <= BASE_JM_MAX_NR_SLOTS );
+
+ atom->retry_submit_on_slot = js;
+}
+
+#if OSK_DISABLE_ASSERTS == 0
+/**
+ * Debug Check the refcount of a context. Only use within ASSERTs
+ *
+ * Obtains kbasep_js_device_data::runpool_irq::lock
+ *
+ * @return negative value if the context is not scheduled in
+ * @return current refcount of the context if it is scheduled in. The refcount
+ * is not guarenteed to be kept constant.
+ */
+static INLINE int kbasep_js_debug_check_ctx_refcount( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ int result = -1;
+ int as_nr;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+ js_devdata = &kbdev->js_data;
+
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+ as_nr = kctx->as_nr;
+ if ( as_nr != KBASEP_AS_NR_INVALID )
+ {
+ result = js_devdata->runpool_irq.per_as_data[as_nr].as_busy_refcount;
+ }
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+
+ return result;
+}
+#endif /* OSK_DISABLE_ASSERTS == 0 */
+
+/**
+ * @brief Variant of kbasep_js_runpool_lookup_ctx() that can be used when the
+ * context is guarenteed to be already previously retained.
+ *
+ * It is a programming error to supply the \a as_nr of a context that has not
+ * been previously retained/has a busy refcount of zero. The only exception is
+ * when there is no ctx in \a as_nr (NULL returned).
+ *
+ * The following locking conditions are made on the caller:
+ * - it must \em not hold the kbasep_js_device_data::runpoool_irq::lock, because
+ * it will be used internally.
+ *
+ * @return a valid kbase_context on success, with a refcount that is guarenteed
+ * to be non-zero and unmodified by this function.
+ * @return NULL on failure, indicating that no context was found in \a as_nr
+ */
+static INLINE kbase_context* kbasep_js_runpool_lookup_ctx_noretain( kbase_device *kbdev, int as_nr )
+{
+ kbasep_js_device_data *js_devdata;
+ kbase_context *found_kctx;
+ kbasep_js_per_as_data *js_per_as_data;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( 0 <= as_nr && as_nr < BASE_MAX_NR_AS );
+ js_devdata = &kbdev->js_data;
+ js_per_as_data = &js_devdata->runpool_irq.per_as_data[as_nr];
+
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+
+ found_kctx = js_per_as_data->kctx;
+ OSK_ASSERT( found_kctx == NULL || js_per_as_data->as_busy_refcount > 0 );
+
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+
+ return found_kctx;
+}
+
+
+/**
+ * @note MIDBASE-769: OSK to add high resolution timer
+ */
+static INLINE kbasep_js_tick kbasep_js_get_js_ticks( void )
+{
+ return osk_time_now();
+}
+
+/**
+ * Supports about an hour worth of time difference, allows the underlying
+ * clock to be more/less accurate than microseconds
+ *
+ * @note MIDBASE-769: OSK to add high resolution timer
+ */
+static INLINE u32 kbasep_js_convert_js_ticks_to_us( kbasep_js_tick js_tick )
+{
+ return (js_tick * 10000u) / osk_time_mstoticks(10u);
+}
+
+/**
+ * Supports about an hour worth of time difference, allows the underlying
+ * clock to be more/less accurate than microseconds
+ *
+ * @note MIDBASE-769: OSK to add high resolution timer
+ */
+static INLINE kbasep_js_tick kbasep_js_convert_js_us_to_ticks( u32 us )
+{
+ return (us * (kbasep_js_tick)osk_time_mstoticks(1000u)) / 1000000u;
+}
+/**
+ * Determine if ticka comes after tickb
+ *
+ * @note MIDBASE-769: OSK to add high resolution timer
+ */
+static INLINE mali_bool kbasep_js_ticks_after( kbasep_js_tick ticka, kbasep_js_tick tickb )
+{
+ kbasep_js_tick tick_diff = ticka - tickb;
+ const kbasep_js_tick wrapvalue = ((kbasep_js_tick)1u) << ((sizeof(kbasep_js_tick)*8)-1);
+
+ return (mali_bool)(tick_diff < wrapvalue);
+}
+
+/**
+ * This will provide a conversion from time (us) to ticks of the gpu clock
+ * based on the minimum available gpu frequency.
+ * This is usually good to compute best/worst case (where the use of current
+ * frequency is not valid due to DVFS).
+ * e.g.: when you need the number of cycles to guarantee you won't wait for
+ * longer than 'us' time (you might have a shorter wait).
+ */
+static INLINE kbasep_js_gpu_tick kbasep_js_convert_us_to_gpu_ticks_min_freq( kbase_device *kbdev, u32 us )
+{
+ u32 gpu_freq = kbdev->gpu_props.props.core_props.gpu_freq_khz_min;
+ OSK_ASSERT( 0!= gpu_freq );
+ return (us * (gpu_freq / 1000));
+}
+
+/**
+ * This will provide a conversion from time (us) to ticks of the gpu clock
+ * based on the maximum available gpu frequency.
+ * This is usually good to compute best/worst case (where the use of current
+ * frequency is not valid due to DVFS).
+ * e.g.: When you need the number of cycles to guarantee you'll wait at least
+ * 'us' amount of time (but you might wait longer).
+ */
+static INLINE kbasep_js_gpu_tick kbasep_js_convert_us_to_gpu_ticks_max_freq( kbase_device *kbdev, u32 us )
+{
+ u32 gpu_freq = kbdev->gpu_props.props.core_props.gpu_freq_khz_max;
+ OSK_ASSERT( 0!= gpu_freq );
+ return (us * (kbasep_js_gpu_tick)(gpu_freq / 1000));
+}
+
+/**
+ * This will provide a conversion from ticks of the gpu clock to time (us)
+ * based on the minimum available gpu frequency.
+ * This is usually good to compute best/worst case (where the use of current
+ * frequency is not valid due to DVFS).
+ * e.g.: When you need to know the worst-case wait that 'ticks' cycles will
+ * take (you guarantee that you won't wait any longer than this, but it may
+ * be shorter).
+ */
+static INLINE u32 kbasep_js_convert_gpu_ticks_to_us_min_freq( kbase_device *kbdev, kbasep_js_gpu_tick ticks )
+{
+ u32 gpu_freq = kbdev->gpu_props.props.core_props.gpu_freq_khz_min;
+ OSK_ASSERT( 0!= gpu_freq );
+ return (ticks / gpu_freq * 1000);
+}
+
+/**
+ * This will provide a conversion from ticks of the gpu clock to time (us)
+ * based on the maximum available gpu frequency.
+ * This is usually good to compute best/worst case (where the use of current
+ * frequency is not valid due to DVFS).
+ * e.g.: When you need to know the best-case wait for 'tick' cycles (you
+ * guarantee to be waiting for at least this long, but it may be longer).
+ */
+static INLINE u32 kbasep_js_convert_gpu_ticks_to_us_max_freq( kbase_device *kbdev, kbasep_js_gpu_tick ticks )
+{
+ u32 gpu_freq = kbdev->gpu_props.props.core_props.gpu_freq_khz_max;
+ OSK_ASSERT( 0!= gpu_freq );
+ return (ticks / gpu_freq * 1000);
+}
+/** @} */ /* end group kbase_js */
+/** @} */ /* end group base_kbase_api */
+/** @} */ /* end group base_api */
+
+#endif /* _KBASE_JS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_js_affinity.c
+ * Base kernel affinity manager APIs
+ */
+
+#include <kbase/src/common/mali_kbase.h>
+#include "mali_kbase_js_affinity.h"
+
+#if MALI_DEBUG && 0 /* disabled to avoid compilation warnings */
+
+STATIC void debug_get_binary_string(const u64 n, char *buff, const int size)
+{
+ unsigned int i;
+ for (i = 0; i < size; i++)
+ {
+ buff[i] = ((n >> i) & 1) ? '*' : '-';
+ }
+ buff[size] = '\0';
+}
+
+#define N_CORES 8
+STATIC void debug_print_affinity_info(const kbase_device *kbdev, const kbase_jd_atom *katom, int js, u64 affinity)
+{
+ char buff[N_CORES +1];
+ char buff2[N_CORES +1];
+ base_jd_core_req core_req = katom->atom->core_req;
+ u8 nr_nss_ctxs_running = kbdev->js_data.runpool_irq.ctx_attr_ref_count[KBASEP_JS_CTX_ATTR_NSS];
+ u64 shader_present_bitmap = kbdev->shader_present_bitmap;
+
+ debug_get_binary_string(shader_present_bitmap, buff, N_CORES);
+ debug_get_binary_string(affinity, buff2, N_CORES);
+
+ OSK_PRINT_INFO(OSK_BASE_JM, "Job: NSS COH FS CS T CF V JS | NSS_ctx | GPU:12345678 | AFF:12345678");
+ OSK_PRINT_INFO(OSK_BASE_JM, " %s %s %s %s %s %s %s %u | %u | %s | %s",
+ core_req & BASE_JD_REQ_NSS ? "*" : "-",
+ core_req & BASE_JD_REQ_COHERENT_GROUP ? "*" : "-",
+ core_req & BASE_JD_REQ_FS ? "*" : "-",
+ core_req & BASE_JD_REQ_CS ? "*" : "-",
+ core_req & BASE_JD_REQ_T ? "*" : "-",
+ core_req & BASE_JD_REQ_CF ? "*" : "-",
+ core_req & BASE_JD_REQ_V ? "*" : "-",
+ js, nr_nss_ctxs_running, buff, buff2);
+}
+
+#endif /* MALI_DEBUG */
+
+OSK_STATIC_INLINE mali_bool affinity_job_uses_high_cores( kbase_device *kbdev, kbase_jd_atom *katom )
+{
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8987))
+ {
+ kbase_context *kctx;
+ kbase_context_flags ctx_flags;
+
+ kctx = katom->kctx;
+ ctx_flags = kctx->jctx.sched_info.ctx.flags;
+
+ /* In this HW Workaround, compute-only jobs/contexts use the high cores
+ * during a core-split, all other contexts use the low cores. */
+ return (mali_bool)((katom->core_req & BASE_JD_REQ_ONLY_COMPUTE) != 0
+ || (ctx_flags & KBASE_CTX_FLAG_HINT_ONLY_COMPUTE) != 0);
+ }
+ else
+ {
+ base_jd_core_req core_req = katom->core_req;
+ /* NSS-ness determines whether the high cores in a core split are used */
+ return (mali_bool)(core_req & BASE_JD_REQ_NSS);
+ }
+}
+
+
+/**
+ * @brief Decide whether a split in core affinity is required across job slots
+ *
+ * The following locking conditions are made on the caller:
+ * - it must hold kbasep_js_device_data::runpool_irq::lock
+ *
+ * @param kbdev The kbase device structure of the device
+ * @return MALI_FALSE if a core split is not required
+ * @return != MALI_FALSE if a core split is required.
+ */
+OSK_STATIC_INLINE mali_bool kbase_affinity_requires_split(kbase_device *kbdev)
+{
+ kbasep_js_device_data *js_devdata;
+
+ OSK_ASSERT( kbdev != NULL );
+ js_devdata = &kbdev->js_data;
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8987))
+ {
+ s8 nr_compute_ctxs = kbasep_js_ctx_attr_count_on_runpool( kbdev, KBASEP_JS_CTX_ATTR_COMPUTE );
+ s8 nr_noncompute_ctxs = kbasep_js_ctx_attr_count_on_runpool( kbdev, KBASEP_JS_CTX_ATTR_NON_COMPUTE );
+
+ /* In this case, a mix of Compute+Non-Compute determines whether a
+ * core-split is required, to ensure jobs with different numbers of RMUs
+ * don't use the same cores.
+ *
+ * When it's entirely compute, or entirely non-compute, then no split is
+ * required.
+ *
+ * A context can be both Compute and Non-compute, in which case this will
+ * correctly decide that a core-split is required. */
+
+ return (mali_bool)( nr_compute_ctxs > 0 && nr_noncompute_ctxs > 0 );
+ }
+ else
+ {
+ /* NSS/SS state determines whether a core-split is required */
+ return kbasep_js_ctx_attr_is_attr_on_runpool( kbdev, KBASEP_JS_CTX_ATTR_NSS );
+ }
+}
+
+
+
+
+mali_bool kbase_js_can_run_job_on_slot_no_lock( kbase_device *kbdev, int js )
+{
+ /*
+ * Here are the reasons for using job slot 2:
+ * - BASE_HW_ISSUE_8987 (which is entirely used for that purpose)
+ * - NSS atoms (in NSS state, this is entirely used for that)
+ * - In absence of the above two, then:
+ * - Atoms with BASE_JD_REQ_COHERENT_GROUP
+ * - But, only when there aren't contexts with
+ * KBASEP_JS_CTX_ATTR_COMPUTE_ALL_CORES, because the atoms that run on
+ * all cores on slot 1 could be blocked by those using a coherent group
+ * on slot 2
+ * - And, only when you actually have 2 or more coregroups - if you only
+ * have 1 coregroup, then having jobs for slot 2 implies they'd also be
+ * for slot 1, meaning you'll get interference from them. Jobs able to
+ * run on slot 2 could also block jobs that can only run on slot 1
+ * (tiler jobs)
+ */
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8987))
+ {
+ return MALI_TRUE;
+ }
+
+ if ( js != 2 )
+ {
+ return MALI_TRUE;
+ }
+
+ /* Only deal with js==2 now: */
+ if ( kbasep_js_ctx_attr_is_attr_on_runpool( kbdev, KBASEP_JS_CTX_ATTR_NSS ) != MALI_FALSE )
+ {
+ /* In NSS state, slot 2 is used, and exclusively for NSS jobs (which cause a coresplit) */
+ return MALI_TRUE;
+ }
+
+ if ( kbdev->gpu_props.num_core_groups > 1 )
+ {
+ /* Otherwise, only use slot 2 in the 2+ coregroup case */
+ if ( kbasep_js_ctx_attr_is_attr_on_runpool( kbdev, KBASEP_JS_CTX_ATTR_COMPUTE_ALL_CORES ) == MALI_FALSE )
+ {
+ /* ...But only when we *don't* have atoms that run on all cores */
+
+ /* No specific check for BASE_JD_REQ_COHERENT_GROUP atoms - the policy will sort that out */
+ return MALI_TRUE;
+ }
+ }
+
+ /* Above checks failed mean we shouldn't use slot 2 */
+ return MALI_FALSE;
+}
+
+/*
+ * As long as it has been decided to have a deeper modification of
+ * what job scheduler, power manager and affinity manager will
+ * implement, this function is just an intermediate step that
+ * assumes:
+ * - all working cores will be powered on when this is called.
+ * - largest current configuration is a T658 (2x4 cores).
+ * - It has been decided not to have hardcoded values so the low
+ * and high cores in a core split will be evently distributed.
+ * - Odd combinations of core requirements have been filtered out
+ * and do not get to this function (e.g. CS+T+NSS is not
+ * supported here).
+ * - This function is frequently called and can be optimized,
+ * (see notes in loops), but as the functionallity will likely
+ * be modified, optimization has not been addressed.
+*/
+void kbase_js_choose_affinity(u64 *affinity, kbase_device *kbdev, kbase_jd_atom *katom, int js)
+{
+ base_jd_core_req core_req = katom->core_req;
+ u64 shader_present_bitmap = kbdev->shader_present_bitmap;
+ unsigned int num_core_groups = kbdev->gpu_props.num_core_groups;
+
+ OSK_ASSERT(0 != shader_present_bitmap);
+ OSK_ASSERT( js >= 0 );
+
+ if (1 == kbdev->gpu_props.num_cores)
+ {
+ /* trivial case only one core, nothing to do */
+ *affinity = shader_present_bitmap;
+ }
+ else if ( kbase_affinity_requires_split( kbdev ) == MALI_FALSE )
+ {
+ if ( (core_req & (BASE_JD_REQ_COHERENT_GROUP | BASE_JD_REQ_SPECIFIC_COHERENT_GROUP)) )
+ {
+ if ( js == 0 || num_core_groups == 1 )
+ {
+ /* js[0] and single-core-group systems just get the first core group */
+ *affinity = kbdev->gpu_props.props.coherency_info.group[0].core_mask;
+ }
+ else
+ {
+ /* js[1], js[2] use core groups 0, 1 for dual-core-group systems */
+ u32 core_group_idx = ((u32)js) - 1;
+ OSK_ASSERT( core_group_idx < num_core_groups );
+ *affinity = kbdev->gpu_props.props.coherency_info.group[core_group_idx].core_mask;
+ }
+ }
+ else
+ {
+ /* All cores are available when no core split is required */
+ *affinity = shader_present_bitmap;
+ }
+ }
+ else
+ {
+ /* Core split required - divide cores in two non-overlapping groups */
+ u64 low_bitmap, high_bitmap;
+ int n_high_cores = kbdev->gpu_props.num_cores >> 1;
+ OSK_ASSERT(0 != n_high_cores);
+
+ /* compute the reserved high cores bitmap */
+ high_bitmap = ~0;
+ /* note: this can take a while, optimization desirable */
+ while (n_high_cores != osk_count_set_bits(high_bitmap & shader_present_bitmap))
+ {
+ high_bitmap = high_bitmap << 1;
+ }
+ high_bitmap &= shader_present_bitmap;
+
+ /* now decide 4 different situations depending on the low or high
+ * set of cores and requiring coherent group or not */
+ if (affinity_job_uses_high_cores( kbdev, katom ))
+ {
+ OSK_ASSERT(0 != num_core_groups);
+
+ if ( (core_req & (BASE_JD_REQ_COHERENT_GROUP | BASE_JD_REQ_SPECIFIC_COHERENT_GROUP))
+ && (1 != num_core_groups))
+ {
+ /* high set of cores requiring coherency and coherency matters
+ * because we got more than one core group */
+ u64 group1_mask = kbdev->gpu_props.props.coherency_info.group[1].core_mask;
+ *affinity = high_bitmap & group1_mask;
+ }
+ else
+ {
+ /* high set of cores not requiring coherency or coherency is
+ assured as we only have one core_group */
+ *affinity = high_bitmap;
+ }
+ }
+ else
+ {
+ low_bitmap = shader_present_bitmap ^ high_bitmap;
+
+ if ( core_req & (BASE_JD_REQ_COHERENT_GROUP | BASE_JD_REQ_SPECIFIC_COHERENT_GROUP))
+ {
+ /* low set of cores and req coherent group */
+ u64 group0_mask = kbdev->gpu_props.props.coherency_info.group[0].core_mask;
+ u64 low_coh_bitmap = low_bitmap & group0_mask;
+ *affinity = low_coh_bitmap;
+ }
+ else
+ {
+ /* low set of cores and does not req coherent group */
+ *affinity = low_bitmap;
+ }
+ }
+ }
+
+ OSK_ASSERT(*affinity != 0);
+}
+
+OSK_STATIC_INLINE mali_bool kbase_js_affinity_is_violating( kbase_device *kbdev, u64 *affinities )
+{
+ /* This implementation checks whether:
+ * - the two slots involved in Generic thread creation have intersecting affinity
+ * - Cores for the fragment slot (slot 0) would compete with cores for slot 2 when NSS atoms are in use.
+ * - This is due to micro-architectural issues where a job in slot A targetting
+ * cores used by slot B could prevent the job in slot B from making progress
+ * until the job in slot A has completed.
+ * - In our case, when slot 2 is used for batch/NSS atoms, the affinity
+ * intersecting with slot 0 would cause fragment atoms to be delayed by the batch/NSS
+ * atoms.
+ *
+ * @note It just so happens that these restrictions also allow
+ * BASE_HW_ISSUE_8987 to be worked around by placing on job slot 2 the
+ * atoms from ctxs with KBASE_CTX_FLAG_HINT_ONLY_COMPUTE flag set
+ */
+ u64 affinity_set_left;
+ u64 affinity_set_right;
+ u64 intersection;
+ OSK_ASSERT( affinities != NULL );
+
+ affinity_set_left = affinities[1];
+
+ if ( kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8987)
+ || kbasep_js_ctx_attr_is_attr_on_runpool(kbdev, KBASEP_JS_CTX_ATTR_NSS) != MALI_FALSE )
+ {
+ /* The left set also includes those on the Fragment slot when:
+ * - We are using the HW workaround for BASE_HW_ISSUE_8987
+ * - We're in NSS state - to prevent NSS atoms using the same cores as Fragment atoms */
+ affinity_set_left |= affinities[0];
+ }
+
+ affinity_set_right = affinities[2];
+
+ /* A violation occurs when any bit in the left_set is also in the right_set */
+ intersection = affinity_set_left & affinity_set_right;
+
+ return (mali_bool)( intersection != (u64)0u );
+}
+
+mali_bool kbase_js_affinity_would_violate( kbase_device *kbdev, int js, u64 affinity )
+{
+ kbasep_js_device_data *js_devdata;
+ u64 new_affinities[BASE_JM_MAX_NR_SLOTS];
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( js < BASE_JM_MAX_NR_SLOTS );
+ js_devdata = &kbdev->js_data;
+
+ OSK_MEMCPY( new_affinities, js_devdata->runpool_irq.slot_affinities, sizeof(js_devdata->runpool_irq.slot_affinities) );
+
+ new_affinities[ js ] |= affinity;
+
+ return kbase_js_affinity_is_violating( kbdev, new_affinities );
+}
+
+void kbase_js_affinity_retain_slot_cores( kbase_device *kbdev, int js, u64 affinity )
+{
+ kbasep_js_device_data *js_devdata;
+ u64 cores;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( js < BASE_JM_MAX_NR_SLOTS );
+ js_devdata = &kbdev->js_data;
+
+ OSK_ASSERT( kbase_js_affinity_would_violate( kbdev, js, affinity ) == MALI_FALSE );
+
+ cores = affinity;
+ while (cores)
+ {
+ int bitnum = 63 - osk_clz_64(cores);
+ u64 bit = 1ULL << bitnum;
+ s8 cnt;
+
+ OSK_ASSERT( js_devdata->runpool_irq.slot_affinity_refcount[ js ][bitnum] < BASE_JM_SUBMIT_SLOTS );
+
+ cnt = ++(js_devdata->runpool_irq.slot_affinity_refcount[ js ][bitnum]);
+
+ if ( cnt == 1 )
+ {
+ js_devdata->runpool_irq.slot_affinities[js] |= bit;
+ }
+
+ cores &= ~bit;
+ }
+
+}
+
+void kbase_js_affinity_release_slot_cores( kbase_device *kbdev, int js, u64 affinity )
+{
+ kbasep_js_device_data *js_devdata;
+ u64 cores;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( js < BASE_JM_MAX_NR_SLOTS );
+ js_devdata = &kbdev->js_data;
+
+ cores = affinity;
+ while (cores)
+ {
+ int bitnum = 63 - osk_clz_64(cores);
+ u64 bit = 1ULL << bitnum;
+ s8 cnt;
+
+ OSK_ASSERT( js_devdata->runpool_irq.slot_affinity_refcount[ js ][bitnum] > 0 );
+
+ cnt = --(js_devdata->runpool_irq.slot_affinity_refcount[ js ][bitnum]);
+
+ if (0 == cnt)
+ {
+ js_devdata->runpool_irq.slot_affinities[js] &= ~bit;
+ }
+
+ cores &= ~bit;
+ }
+
+}
+
+void kbase_js_affinity_slot_blocked_an_atom( kbase_device *kbdev, int js )
+{
+ kbasep_js_device_data *js_devdata;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( js < BASE_JM_MAX_NR_SLOTS );
+ js_devdata = &kbdev->js_data;
+
+ js_devdata->runpool_irq.slots_blocked_on_affinity |= 1u << js;
+}
+
+void kbase_js_affinity_submit_to_blocked_slots( kbase_device *kbdev )
+{
+ kbasep_js_device_data *js_devdata;
+ u16 slots;
+
+ OSK_ASSERT( kbdev != NULL );
+ js_devdata = &kbdev->js_data;
+
+ osk_mutex_lock( &js_devdata->runpool_mutex );
+
+ /* Must take a copy because submitting jobs will update this member
+ * We don't take a lock here - a data barrier was issued beforehand */
+ slots = js_devdata->runpool_irq.slots_blocked_on_affinity;
+ while (slots)
+ {
+ int bitnum = 31 - osk_clz(slots);
+ u16 bit = 1u << bitnum;
+ slots &= ~bit;
+
+ KBASE_TRACE_ADD_SLOT( kbdev, JS_AFFINITY_SUBMIT_TO_BLOCKED, NULL, NULL, 0u, bitnum);
+
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+ /* must update this before we submit, incase it's set again */
+ js_devdata->runpool_irq.slots_blocked_on_affinity &= ~bit;
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+
+ kbasep_js_try_run_next_job_on_slot( kbdev, bitnum );
+
+ /* Don't re-read slots_blocked_on_affinity after this - it could loop for a long time */
+ }
+ osk_mutex_unlock( &js_devdata->runpool_mutex );
+
+}
+
+#if MALI_DEBUG != 0
+void kbase_js_debug_log_current_affinities( kbase_device *kbdev )
+{
+ kbasep_js_device_data *js_devdata;
+ int slot_nr;
+
+ OSK_ASSERT( kbdev != NULL );
+ js_devdata = &kbdev->js_data;
+
+ for ( slot_nr = 0; slot_nr < 3 ; ++slot_nr )
+ {
+ KBASE_TRACE_ADD_SLOT_INFO( kbdev, JS_AFFINITY_CURRENT, NULL, NULL, 0u, slot_nr, (u32)js_devdata->runpool_irq.slot_affinities[slot_nr] );
+ }
+}
+#endif /* MALI_DEBUG != 0 */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_js_affinity.h
+ * Affinity Manager internal APIs.
+ */
+
+#ifndef _KBASE_JS_AFFINITY_H_
+#define _KBASE_JS_AFFINITY_H_
+
+
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_kbase_api
+ * @{
+ */
+
+
+/**
+ * @addtogroup kbase_js_affinity Affinity Manager internal APIs.
+ * @{
+ *
+ */
+
+
+/**
+ * @brief Decide whether it is possible to submit a job to a particular job slot in the current status
+ *
+ * Will check if submitting to the given job slot is allowed in the current
+ * status. For example using job slot 2 while in soft-stoppable state and only
+ * having 1 coregroup is not allowed by the policy. This function should be
+ * called prior to submitting a job to a slot to make sure policy rules are not
+ * violated.
+ *
+ * The following locking conditions are made on the caller:
+ * - it must hold kbasep_js_device_data::runpool_irq::lock
+ *
+ * @param kbdev The kbase device structure of the device
+ * @param js Job slot number to check for allowance
+ */
+mali_bool kbase_js_can_run_job_on_slot_no_lock( kbase_device *kbdev, int js );
+
+/**
+ * @brief Compute affinity for a given job.
+ *
+ * Currently assumes an all-on/all-off power management policy.
+ * Also assumes there is at least one core with tiler available.
+ * Will try to produce an even distribution of cores for SS and
+ * NSS jobs. SS jobs will be given cores starting from core-group
+ * 0 forward to n. NSS jobs will be given cores from core-group n
+ * backwards to 0. This way for example in a T658 SS jobs will
+ * tend to run on cores from core-group 0 and NSS jobs will tend
+ * to run on cores from core-group 1.
+ * An assertion will be raised if computed affinity is 0
+ *
+ * @param[out] affinity Affinity bitmap computed
+ * @param kbdev The kbase device structure of the device
+ * @param katom Job chain of which affinity is going to be found
+ * @param js Slot the job chain is being submitted
+
+ */
+void kbase_js_choose_affinity( u64 *affinity, kbase_device *kbdev, kbase_jd_atom *katom, int js );
+
+/**
+ * @brief Determine whether a proposed \a affinity on job slot \a js would
+ * cause a violation of affinity restrictions.
+ *
+ * The following locks must be held by the caller:
+ * - kbasep_js_device_data::runpool_irq::lock
+ */
+mali_bool kbase_js_affinity_would_violate( kbase_device *kbdev, int js, u64 affinity );
+
+/**
+ * @brief Affinity tracking: retain cores used by a slot
+ *
+ * The following locks must be held by the caller:
+ * - kbasep_js_device_data::runpool_irq::lock
+ */
+void kbase_js_affinity_retain_slot_cores( kbase_device *kbdev, int js, u64 affinity );
+
+/**
+ * @brief Affinity tracking: release cores used by a slot
+ *
+ * Cores \b must be released as soon as a job is dequeued from a slot's 'submit
+ * slots', and before another job is submitted to those slots. Otherwise, the
+ * refcount could exceed the maximum number submittable to a slot,
+ * BASE_JM_SUBMIT_SLOTS.
+ *
+ * The following locks must be held by the caller:
+ * - kbasep_js_device_data::runpool_irq::lock
+ */
+void kbase_js_affinity_release_slot_cores( kbase_device *kbdev, int js, u64 affinity );
+
+/**
+ * @brief Register a slot as blocking atoms due to affinity violations
+ *
+ * Once a slot has been registered, we must check after every atom completion
+ * (including those on different slots) to see if the slot can be
+ * unblocked. This is done by calling
+ * kbase_js_affinity_submit_to_blocked_slots(), which will also deregister the
+ * slot if it no long blocks atoms due to affinity violations.
+ *
+ * The following locks must be held by the caller:
+ * - kbasep_js_device_data::runpool_irq::lock
+ */
+void kbase_js_affinity_slot_blocked_an_atom( kbase_device *kbdev, int js );
+
+/**
+ * @brief Submit to job slots that have registered that an atom was blocked on
+ * the slot previously due to affinity violations.
+ *
+ * This submits to all slots registered by
+ * kbase_js_affinity_slot_blocked_an_atom(). If submission succeeded, then the
+ * slot is deregistered as having blocked atoms due to affinity
+ * violations. Otherwise it stays registered, and the next atom to complete
+ * must attempt to submit to the blocked slots again.
+ *
+ * The following locking conditions are made on the caller:
+ * - it must \em not hold kbasep_js_device_data::runpool_mutex (as this will be
+ * obtained internally)
+ * - it must \em not hold kbdev->jm_slots[ \a js ].lock (as this will be
+ * obtained internally)
+ * - it must \em not hold kbasep_js_device_data::runpool_irq::lock, (as this will be
+ * obtained internally)
+ */
+void kbase_js_affinity_submit_to_blocked_slots( kbase_device *kbdev );
+
+/**
+ * @brief Output to the Trace log the current tracked affinities on all slots
+ */
+#if MALI_DEBUG != 0
+void kbase_js_debug_log_current_affinities( kbase_device *kbdev );
+#else /* MALI_DEBUG != 0 */
+OSK_STATIC_INLINE void kbase_js_debug_log_current_affinities( kbase_device *kbdev )
+{
+}
+#endif /* MALI_DEBUG != 0 */
+
+/** @} */ /* end group kbase_js_affinity */
+/** @} */ /* end group base_kbase_api */
+/** @} */ /* end group base_api */
+
+
+
+
+
+#endif /* _KBASE_JS_AFFINITY_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+#include <kbase/src/common/mali_kbase.h>
+
+/*
+ * Private functions follow
+ */
+
+/**
+ * @brief Check whether a ctx has a certain attribute, and if so, retain that
+ * attribute on the runpool.
+ *
+ * Requires:
+ * - jsctx mutex
+ * - runpool_irq spinlock
+ * - ctx is scheduled on the runpool
+ *
+ * @return MALI_TRUE indicates a change in ctx attributes state of the runpool.
+ * In this state, the scheduler might be able to submit more jobs than
+ * previously, and so the caller should ensure kbasep_js_try_run_next_job() is
+ * called sometime later.
+ * @return MALI_FALSE indicates no change in ctx attributes state of the runpool.
+ */
+STATIC mali_bool kbasep_js_ctx_attr_runpool_retain_attr( kbase_device *kbdev,
+ kbase_context *kctx,
+ kbasep_js_ctx_attr attribute )
+{
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_kctx_info *js_kctx_info;
+ mali_bool runpool_state_changed = MALI_FALSE;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+ OSK_ASSERT( attribute < KBASEP_JS_CTX_ATTR_COUNT );
+ js_devdata = &kbdev->js_data;
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ OSK_ASSERT( js_kctx_info->ctx.is_scheduled != MALI_FALSE );
+
+ if ( kbasep_js_ctx_attr_is_attr_on_ctx( kctx, attribute ) != MALI_FALSE )
+ {
+ OSK_ASSERT( js_devdata->runpool_irq.ctx_attr_ref_count[attribute] < S8_MAX );
+ ++(js_devdata->runpool_irq.ctx_attr_ref_count[attribute]);
+
+ if ( js_devdata->runpool_irq.ctx_attr_ref_count[attribute] == 1 )
+ {
+ /* First refcount indicates a state change */
+ runpool_state_changed = MALI_TRUE;
+ KBASE_TRACE_ADD( kbdev, JS_CTX_ATTR_NOW_ON_RUNPOOL, kctx, NULL, 0u, attribute );
+ }
+ }
+
+ return runpool_state_changed;
+}
+
+/**
+ * @brief Check whether a ctx has a certain attribute, and if so, release that
+ * attribute on the runpool.
+ *
+ * Requires:
+ * - jsctx mutex
+ * - runpool_irq spinlock
+ * - ctx is scheduled on the runpool
+ *
+ * @return MALI_TRUE indicates a change in ctx attributes state of the runpool.
+ * In this state, the scheduler might be able to submit more jobs than
+ * previously, and so the caller should ensure kbasep_js_try_run_next_job() is
+ * called sometime later.
+ * @return MALI_FALSE indicates no change in ctx attributes state of the runpool.
+ */
+STATIC mali_bool kbasep_js_ctx_attr_runpool_release_attr( kbase_device *kbdev,
+ kbase_context *kctx,
+ kbasep_js_ctx_attr attribute )
+{
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_kctx_info *js_kctx_info;
+ mali_bool runpool_state_changed = MALI_FALSE;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+ OSK_ASSERT( attribute < KBASEP_JS_CTX_ATTR_COUNT );
+ js_devdata = &kbdev->js_data;
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ OSK_ASSERT( js_kctx_info->ctx.is_scheduled != MALI_FALSE );
+
+ if ( kbasep_js_ctx_attr_is_attr_on_ctx( kctx, attribute ) != MALI_FALSE )
+ {
+ OSK_ASSERT( js_devdata->runpool_irq.ctx_attr_ref_count[attribute] > 0 );
+ --(js_devdata->runpool_irq.ctx_attr_ref_count[attribute]);
+
+ if ( js_devdata->runpool_irq.ctx_attr_ref_count[attribute] == 0 )
+ {
+ /* Last de-refcount indicates a state change */
+ runpool_state_changed = MALI_TRUE;
+ KBASE_TRACE_ADD( kbdev, JS_CTX_ATTR_NOW_OFF_RUNPOOL, kctx, NULL, 0u, attribute );
+ }
+ }
+
+ return runpool_state_changed;
+}
+
+/**
+ * @brief Retain a certain attribute on a ctx, also retaining it on the runpool
+ * if the context is scheduled.
+ *
+ * Requires:
+ * - jsctx mutex
+ * - If the context is scheduled, then runpool_irq spinlock must also be held
+ *
+ * @return MALI_TRUE indicates a change in ctx attributes state of the runpool.
+ * This may allow the scheduler to submit more jobs than previously.
+ * @return MALI_FALSE indicates no change in ctx attributes state of the runpool.
+ */
+STATIC mali_bool kbasep_js_ctx_attr_ctx_retain_attr( kbase_device *kbdev,
+ kbase_context *kctx,
+ kbasep_js_ctx_attr attribute )
+{
+ kbasep_js_kctx_info *js_kctx_info;
+ mali_bool runpool_state_changed = MALI_FALSE;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+ OSK_ASSERT( attribute < KBASEP_JS_CTX_ATTR_COUNT );
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ OSK_ASSERT( js_kctx_info->ctx.ctx_attr_ref_count[ attribute ] < U32_MAX );
+
+ ++(js_kctx_info->ctx.ctx_attr_ref_count[ attribute ]);
+
+ if ( js_kctx_info->ctx.is_scheduled != MALI_FALSE
+ && js_kctx_info->ctx.ctx_attr_ref_count[ attribute ] == 1 )
+ {
+ /* Only ref-count the attribute on the runpool for the first time this contexts sees this attribute */
+ KBASE_TRACE_ADD( kbdev, JS_CTX_ATTR_NOW_ON_CTX, kctx, NULL, 0u, attribute );
+ runpool_state_changed = kbasep_js_ctx_attr_runpool_retain_attr( kbdev, kctx, attribute );
+ }
+
+ return runpool_state_changed;
+}
+
+/**
+ * @brief Release a certain attribute on a ctx, also releasign it from the runpool
+ * if the context is scheduled.
+ *
+ * Requires:
+ * - jsctx mutex
+ * - If the context is scheduled, then runpool_irq spinlock must also be held
+ *
+ * @return MALI_TRUE indicates a change in ctx attributes state of the runpool.
+ * This may allow the scheduler to submit more jobs than previously.
+ * @return MALI_FALSE indicates no change in ctx attributes state of the runpool.
+ */
+STATIC mali_bool kbasep_js_ctx_attr_ctx_release_attr( kbase_device *kbdev,
+ kbase_context *kctx,
+ kbasep_js_ctx_attr attribute )
+{
+ kbasep_js_kctx_info *js_kctx_info;
+ mali_bool runpool_state_changed = MALI_FALSE;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+ OSK_ASSERT( attribute < KBASEP_JS_CTX_ATTR_COUNT );
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ OSK_ASSERT( js_kctx_info->ctx.ctx_attr_ref_count[ attribute ] > 0 );
+
+ if ( js_kctx_info->ctx.is_scheduled != MALI_FALSE
+ && js_kctx_info->ctx.ctx_attr_ref_count[ attribute ] == 1 )
+ {
+ /* Only de-ref-count the attribute on the runpool when this is the last ctx-reference to it */
+ runpool_state_changed = kbasep_js_ctx_attr_runpool_release_attr( kbdev, kctx, attribute );
+ KBASE_TRACE_ADD( kbdev, JS_CTX_ATTR_NOW_OFF_CTX, kctx, NULL, 0u, attribute );
+ }
+
+ /* De-ref must happen afterwards, because kbasep_js_ctx_attr_runpool_release() needs to check it too */
+ --(js_kctx_info->ctx.ctx_attr_ref_count[ attribute ]);
+
+ return runpool_state_changed;
+}
+
+
+/*
+ * More commonly used public functions
+ */
+
+void kbasep_js_ctx_attr_set_initial_attrs( kbase_device *kbdev,
+ kbase_context *kctx )
+{
+ kbasep_js_kctx_info *js_kctx_info;
+ mali_bool runpool_state_changed = MALI_FALSE;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ if ( (js_kctx_info->ctx.flags & KBASE_CTX_FLAG_SUBMIT_DISABLED) != MALI_FALSE )
+ {
+ /* This context never submits, so don't track any scheduling attributes */
+ return;
+ }
+
+ /* Transfer attributes held in the context flags for contexts that have submit enabled */
+
+ if ( (js_kctx_info->ctx.flags & KBASE_CTX_FLAG_HINT_ONLY_COMPUTE) != MALI_FALSE )
+ {
+ /* Compute context */
+ runpool_state_changed |= kbasep_js_ctx_attr_ctx_retain_attr( kbdev, kctx, KBASEP_JS_CTX_ATTR_COMPUTE );
+ }
+ /* NOTE: Whether this is a non-compute context depends on the jobs being
+ * run, e.g. it might be submitting jobs with BASE_JD_REQ_ONLY_COMPUTE */
+
+ /* ... More attributes can be added here ... */
+
+ /* The context should not have been scheduled yet, so ASSERT if this caused
+ * runpool state changes (note that other threads *can't* affect the value
+ * of runpool_state_changed, due to how it's calculated) */
+ OSK_ASSERT( runpool_state_changed == MALI_FALSE );
+ CSTD_UNUSED( runpool_state_changed );
+}
+
+void kbasep_js_ctx_attr_runpool_retain_ctx( kbase_device *kbdev,
+ kbase_context *kctx )
+{
+ mali_bool runpool_state_changed;
+ int i;
+
+ /* Retain any existing attributes */
+ for ( i = 0 ; i < KBASEP_JS_CTX_ATTR_COUNT; ++i )
+ {
+ if ( kbasep_js_ctx_attr_is_attr_on_ctx( kctx, (kbasep_js_ctx_attr)i ) != MALI_FALSE )
+ {
+ /* The context is being scheduled in, so update the runpool with the new attributes */
+ runpool_state_changed = kbasep_js_ctx_attr_runpool_retain_attr( kbdev, kctx, (kbasep_js_ctx_attr)i );
+
+ /* We don't need to know about state changed, because retaining a
+ * context occurs on scheduling it, and that itself will also try
+ * to run new atoms */
+ CSTD_UNUSED( runpool_state_changed );
+ }
+ }
+}
+
+mali_bool kbasep_js_ctx_attr_runpool_release_ctx( kbase_device *kbdev,
+ kbase_context *kctx )
+{
+ mali_bool runpool_state_changed = MALI_FALSE;
+ int i;
+
+ /* Release any existing attributes */
+ for ( i = 0 ; i < KBASEP_JS_CTX_ATTR_COUNT; ++i )
+ {
+ if ( kbasep_js_ctx_attr_is_attr_on_ctx( kctx, (kbasep_js_ctx_attr)i ) != MALI_FALSE )
+ {
+ /* The context is being scheduled out, so update the runpool on the removed attributes */
+ runpool_state_changed |= kbasep_js_ctx_attr_runpool_release_attr( kbdev, kctx, (kbasep_js_ctx_attr)i );
+ }
+ }
+
+ return runpool_state_changed;
+}
+
+void kbasep_js_ctx_attr_ctx_retain_atom( kbase_device *kbdev,
+ kbase_context *kctx,
+ kbase_jd_atom *katom )
+{
+ mali_bool runpool_state_changed = MALI_FALSE;
+ base_jd_core_req core_req;
+
+ OSK_ASSERT( katom );
+ core_req = katom->core_req;
+
+ if ( core_req & BASE_JD_REQ_NSS )
+ {
+ runpool_state_changed |= kbasep_js_ctx_attr_ctx_retain_attr( kbdev, kctx, KBASEP_JS_CTX_ATTR_NSS );
+ }
+
+ if ( core_req & BASE_JD_REQ_ONLY_COMPUTE )
+ {
+ runpool_state_changed |= kbasep_js_ctx_attr_ctx_retain_attr( kbdev, kctx, KBASEP_JS_CTX_ATTR_COMPUTE );
+ }
+ else
+ {
+ runpool_state_changed |= kbasep_js_ctx_attr_ctx_retain_attr( kbdev, kctx, KBASEP_JS_CTX_ATTR_NON_COMPUTE );
+ }
+
+ if ( (core_req & ( BASE_JD_REQ_CS | BASE_JD_REQ_ONLY_COMPUTE | BASE_JD_REQ_T)) != 0
+ && (core_req & ( BASE_JD_REQ_COHERENT_GROUP | BASE_JD_REQ_SPECIFIC_COHERENT_GROUP )) == 0 )
+ {
+ /* Atom that can run on slot1 or slot2, and can use all cores */
+ runpool_state_changed |= kbasep_js_ctx_attr_ctx_retain_attr( kbdev, kctx, KBASEP_JS_CTX_ATTR_COMPUTE_ALL_CORES );
+ }
+
+ /* We don't need to know about state changed, because retaining an
+ * atom occurs on adding it, and that itself will also try to run
+ * new atoms */
+ CSTD_UNUSED( runpool_state_changed );
+}
+
+mali_bool kbasep_js_ctx_attr_ctx_release_atom( kbase_device *kbdev,
+ kbase_context *kctx,
+ kbase_jd_atom *katom )
+{
+ mali_bool runpool_state_changed = MALI_FALSE;
+ base_jd_core_req core_req;
+
+ OSK_ASSERT( katom );
+ core_req = katom->core_req;
+
+ if ( core_req & BASE_JD_REQ_NSS )
+ {
+ runpool_state_changed |= kbasep_js_ctx_attr_ctx_release_attr( kbdev, kctx, KBASEP_JS_CTX_ATTR_NSS );
+ }
+
+ if ( core_req & BASE_JD_REQ_ONLY_COMPUTE )
+ {
+ runpool_state_changed |= kbasep_js_ctx_attr_ctx_release_attr( kbdev, kctx, KBASEP_JS_CTX_ATTR_COMPUTE );
+ }
+ else
+ {
+ runpool_state_changed |= kbasep_js_ctx_attr_ctx_release_attr( kbdev, kctx, KBASEP_JS_CTX_ATTR_NON_COMPUTE );
+ }
+
+ if ( (core_req & ( BASE_JD_REQ_CS | BASE_JD_REQ_ONLY_COMPUTE | BASE_JD_REQ_T )) != 0
+ && (core_req & ( BASE_JD_REQ_COHERENT_GROUP | BASE_JD_REQ_SPECIFIC_COHERENT_GROUP )) == 0 )
+ {
+ /* Atom that can run on slot1 or slot2, and can use all cores */
+ runpool_state_changed |= kbasep_js_ctx_attr_ctx_release_attr( kbdev, kctx, KBASEP_JS_CTX_ATTR_COMPUTE_ALL_CORES );
+ }
+
+
+ return runpool_state_changed;
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_js_ctx_attr.h
+ * Job Scheduler Context Attribute APIs
+ */
+
+
+#ifndef _KBASE_JS_CTX_ATTR_H_
+#define _KBASE_JS_CTX_ATTR_H_
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_kbase_api
+ * @{
+ */
+
+/**
+ * @addtogroup kbase_js
+ * @{
+ */
+
+/**
+ * Set the initial attributes of a context (when context create flags are set)
+ *
+ * Requires:
+ * - Hold the jsctx_mutex
+ */
+void kbasep_js_ctx_attr_set_initial_attrs( kbase_device *kbdev,
+ kbase_context *kctx );
+
+/**
+ * Retain all attributes of a context
+ *
+ * This occurs on scheduling in the context on the runpool (but after
+ * is_scheduled is set)
+ *
+ * Requires:
+ * - jsctx mutex
+ * - runpool_irq spinlock
+ * - ctx->is_scheduled is true
+ */
+void kbasep_js_ctx_attr_runpool_retain_ctx( kbase_device *kbdev,
+ kbase_context *kctx );
+
+/**
+ * Release all attributes of a context
+ *
+ * This occurs on scheduling out the context from the runpool (but before
+ * is_scheduled is cleared)
+ *
+ * Requires:
+ * - jsctx mutex
+ * - runpool_irq spinlock
+ * - ctx->is_scheduled is true
+ *
+ * @return MALI_TRUE indicates a change in ctx attributes state of the runpool.
+ * In this state, the scheduler might be able to submit more jobs than
+ * previously, and so the caller should ensure kbasep_js_try_run_next_job() is
+ * called sometime later.
+ * @return MALI_FALSE indicates no change in ctx attributes state of the runpool.
+ */
+mali_bool kbasep_js_ctx_attr_runpool_release_ctx( kbase_device *kbdev,
+ kbase_context *kctx );
+
+/**
+ * Retain all attributes of an atom
+ *
+ * This occurs on adding an atom to a context
+ *
+ * Requires:
+ * - jsctx mutex
+ * - If the context is scheduled, then runpool_irq spinlock must also be held
+ */
+void kbasep_js_ctx_attr_ctx_retain_atom( kbase_device *kbdev,
+ kbase_context *kctx,
+ kbase_jd_atom *katom );
+
+/**
+ * Release all attributes of an atom
+ *
+ * This occurs on (permanently) removing an atom from a context
+ *
+ * Requires:
+ * - jsctx mutex
+ * - If the context is scheduled, then runpool_irq spinlock must also be held
+ *
+ * @return MALI_TRUE indicates a change in ctx attributes state of the runpool.
+ * In this state, the scheduler might be able to submit more jobs than
+ * previously, and so the caller should ensure kbasep_js_try_run_next_job() is
+ * called sometime later.
+ * @return MALI_FALSE indicates no change in ctx attributes state of the runpool.
+ */
+mali_bool kbasep_js_ctx_attr_ctx_release_atom( kbase_device *kbdev,
+ kbase_context *kctx,
+ kbase_jd_atom *katom );
+
+/**
+ * Requires:
+ * - runpool_irq spinlock
+ */
+OSK_STATIC_INLINE s8 kbasep_js_ctx_attr_count_on_runpool( kbase_device *kbdev, kbasep_js_ctx_attr attribute )
+{
+ kbasep_js_device_data *js_devdata;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( attribute < KBASEP_JS_CTX_ATTR_COUNT );
+ js_devdata = &kbdev->js_data;
+
+ return js_devdata->runpool_irq.ctx_attr_ref_count[attribute];
+}
+
+
+/**
+ * Requires:
+ * - runpool_irq spinlock
+ */
+OSK_STATIC_INLINE mali_bool kbasep_js_ctx_attr_is_attr_on_runpool( kbase_device *kbdev, kbasep_js_ctx_attr attribute )
+{
+ /* In general, attributes are 'on' when they have a non-zero refcount (note: the refcount will never be < 0) */
+ return (mali_bool)kbasep_js_ctx_attr_count_on_runpool( kbdev, attribute );
+}
+
+/**
+ * Requires:
+ * - jsctx mutex
+ */
+OSK_STATIC_INLINE mali_bool kbasep_js_ctx_attr_is_attr_on_ctx( kbase_context *kctx, kbasep_js_ctx_attr attribute )
+{
+ kbasep_js_kctx_info *js_kctx_info;
+
+ OSK_ASSERT( kctx != NULL );
+ OSK_ASSERT( attribute < KBASEP_JS_CTX_ATTR_COUNT );
+ js_kctx_info = &kctx->jctx.sched_info;
+
+ /* In general, attributes are 'on' when they have a refcount (which should never be < 0) */
+ return (mali_bool)(js_kctx_info->ctx.ctx_attr_ref_count[ attribute ]);
+}
+
+/** @} */ /* end group kbase_js */
+/** @} */ /* end group base_kbase_api */
+/** @} */ /* end group base_api */
+
+#endif /* _KBASE_JS_DEFS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_js.h
+ * Job Scheduler Type Definitions
+ */
+
+
+#ifndef _KBASE_JS_DEFS_H_
+#define _KBASE_JS_DEFS_H_
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_kbase_api
+ * @{
+ */
+
+/**
+ * @addtogroup kbase_js
+ * @{
+ */
+
+/* Types used by the policies must go here */
+enum
+{
+ /** Context has had its creation flags set */
+ KBASE_CTX_FLAG_CREATE_FLAGS_SET = (1u << 0),
+
+ /** Context will not submit any jobs */
+ KBASE_CTX_FLAG_SUBMIT_DISABLED = (1u << 1),
+
+ /** Set if the context uses an address space and should be kept scheduled in */
+ KBASE_CTX_FLAG_PRIVILEGED = (1u << 2),
+
+ /** Kernel-side equivalent of BASE_CONTEXT_HINT_ONLY_COMPUTE. Non-mutable after creation flags set */
+ KBASE_CTX_FLAG_HINT_ONLY_COMPUTE= (1u << 3)
+
+ /* NOTE: Add flags for other things, such as 'is scheduled', and 'is dying' */
+};
+
+typedef u32 kbase_context_flags;
+
+typedef struct kbasep_atom_req
+{
+ base_jd_core_req core_req;
+ kbase_context_flags ctx_req;
+ u32 device_nr;
+} kbasep_atom_req;
+
+#include "mali_kbase_js_policy_cfs.h"
+
+
+/* Wrapper Interface - doxygen is elsewhere */
+typedef union kbasep_js_policy
+{
+#ifdef KBASE_JS_POLICY_AVAILABLE_FCFS
+ kbasep_js_policy_fcfs fcfs;
+#endif
+#ifdef KBASE_JS_POLICY_AVAILABLE_CFS
+ kbasep_js_policy_cfs cfs;
+#endif
+} kbasep_js_policy;
+
+/* Wrapper Interface - doxygen is elsewhere */
+typedef union kbasep_js_policy_ctx_info
+{
+#ifdef KBASE_JS_POLICY_AVAILABLE_FCFS
+ kbasep_js_policy_fcfs_ctx fcfs;
+#endif
+#ifdef KBASE_JS_POLICY_AVAILABLE_CFS
+ kbasep_js_policy_cfs_ctx cfs;
+#endif
+} kbasep_js_policy_ctx_info;
+
+/* Wrapper Interface - doxygen is elsewhere */
+typedef union kbasep_js_policy_job_info
+{
+#ifdef KBASE_JS_POLICY_AVAILABLE_FCFS
+ kbasep_js_policy_fcfs_job fcfs;
+#endif
+#ifdef KBASE_JS_POLICY_AVAILABLE_CFS
+ kbasep_js_policy_cfs_job cfs;
+#endif
+} kbasep_js_policy_job_info;
+
+/**
+ * @brief Maximum number of jobs that can be submitted to a job slot whilst
+ * inside the IRQ handler.
+ *
+ * This is important because GPU NULL jobs can complete whilst the IRQ handler
+ * is running. Otherwise, it potentially allows an unlimited number of GPU NULL
+ * jobs to be submitted inside the IRQ handler, which increases IRQ latency.
+ */
+#define KBASE_JS_MAX_JOB_SUBMIT_PER_SLOT_PER_IRQ 2
+
+/**
+ * @brief the IRQ_THROTTLE time in microseconds
+ *
+ * This will be converted via the GPU's clock frequency into a cycle-count.
+ *
+ * @note we can make an estimate of the GPU's frequency by periodically
+ * sampling its CYCLE_COUNT register
+ */
+#define KBASE_JS_IRQ_THROTTLE_TIME_US 20
+
+/**
+ * @brief Context attributes
+ *
+ * Each context attribute can be thought of as a boolean value that caches some
+ * state information about either the runpool, or the context:
+ * - In the case of the runpool, it is a cache of "Do any contexts owned by
+ * the runpool have attribute X?"
+ * - In the case of a context, it is a cache of "Do any atoms owned by the
+ * context have attribute X?"
+ *
+ * The boolean value of the context attributes often affect scheduling
+ * decisions, such as affinities to use and job slots to use.
+ *
+ * To accomodate changes of state in the context, each attribute is refcounted
+ * in the context, and in the runpool for all running contexts. Specifically:
+ * - The runpool holds a refcount of how many contexts in the runpool have this
+ * attribute.
+ * - The context holds a refcount of how many atoms have this attribute.
+ *
+ * Examples of use:
+ * - Finding out when NSS jobs are in the runpool
+ * - Finding out when there are a mix of @ref BASE_CONTEXT_HINT_ONLY_COMPUTE
+ * and ! @ref BASE_CONTEXT_HINT_ONLY_COMPUTE contexts in the runpool
+ */
+typedef enum
+{
+ /** Attribute indicating an NSS context */
+ KBASEP_JS_CTX_ATTR_NSS,
+
+ /** Attribute indicating a context that contains Compute jobs. That is,
+ * @ref BASE_CONTEXT_HINT_COMPUTE is \b set and/or the context has jobs of type
+ * @ref BASE_JD_REQ_ONLY_COMPUTE
+ *
+ * @note A context can be both 'Compute' and 'Non Compute' if it contains
+ * both types of jobs.
+ */
+ KBASEP_JS_CTX_ATTR_COMPUTE,
+
+ /** Attribute indicating a context that contains Non-Compute jobs. That is,
+ * the context has some jobs that are \b not of type @ref
+ * BASE_JD_REQ_ONLY_COMPUTE. The context usually has
+ * BASE_CONTEXT_HINT_COMPUTE \b clear, but this depends on the HW
+ * workarounds in use in the Job Scheduling Policy.
+ *
+ * @note A context can be both 'Compute' and 'Non Compute' if it contains
+ * both types of jobs.
+ */
+ KBASEP_JS_CTX_ATTR_NON_COMPUTE,
+
+ /** Attribute indicating that a context contains compute-job atoms that
+ * aren't restricted to a coherent group, and can run on all cores.
+ *
+ * Specifically, this is when the atom's \a core_req satisfy:
+ * - (\a core_req & (BASE_JD_REQ_CS | BASE_JD_REQ_ONLY_COMPUTE | BASE_JD_REQ_T) // uses slot 1 or slot 2
+ * - && !(\a core_req & BASE_JD_REQ_COHERENT_GROUP) // not restricted to coherent groups
+ *
+ * Such atoms could be blocked from running if one of the coherent groups
+ * is being used by another job slot, so tracking this context attribute
+ * allows us to prevent such situations.
+ *
+ * @note This doesn't take into account the 1-coregroup case, where all
+ * compute atoms would effectively be able to run on 'all cores', but
+ * contexts will still not always get marked with this attribute. Instead,
+ * it is the caller's responsibility to take into account the number of
+ * coregroups when interpreting this attribute.
+ *
+ * @note Whilst Tiler atoms are normally combined with
+ * BASE_JD_REQ_COHERENT_GROUP, it is possible to send such atoms without
+ * BASE_JD_REQ_COHERENT_GROUP set. This is an unlikely case, but it's easy
+ * enough to handle anyway.
+ */
+ KBASEP_JS_CTX_ATTR_COMPUTE_ALL_CORES,
+
+ /** Must be the last in the enum */
+ KBASEP_JS_CTX_ATTR_COUNT
+} kbasep_js_ctx_attr;
+
+
+/**
+ * Data used by the scheduler that is unique for each Address Space.
+ *
+ * This is used in IRQ context and kbasep_js_device_data::runpoool_irq::lock
+ * must be held whilst accessing this data (inculding reads and atomic
+ * decisions based on the read).
+ */
+typedef struct kbasep_js_per_as_data
+{
+ /**
+ * Ref count of whether this AS is busy, and must not be scheduled out
+ *
+ * When jobs are running this is always positive. However, it can still be
+ * positive when no jobs are running. If all you need is a heuristic to
+ * tell you whether jobs might be running, this should be sufficient.
+ */
+ int as_busy_refcount;
+
+ /** Pointer to the current context on this address space, or NULL for no context */
+ kbase_context *kctx;
+} kbasep_js_per_as_data;
+
+/**
+ * @brief KBase Device Data Job Scheduler sub-structure
+ *
+ * This encapsulates the current context of the Job Scheduler on a particular
+ * device. This context is global to the device, and is not tied to any
+ * particular kbase_context running on the device.
+ *
+ * nr_contexts_running, nr_nss_ctxs_running and as_free are
+ * optimized for packing together (by making them smaller types than u32). The
+ * operations on them should rarely involve masking. The use of signed types for
+ * arithmetic indicates to the compiler that the value will not rollover (which
+ * would be undefined behavior), and so under the Total License model, it is free
+ * to make optimizations based on that (i.e. to remove masking).
+ */
+typedef struct kbasep_js_device_data
+{
+ /** Sub-structure to collect together Job Scheduling data used in IRQ context */
+ struct
+ {
+ /**
+ * Lock for accessing Job Scheduling data used in IRQ context
+ *
+ * This lock must be held whenever this data is accessed (read, or
+ * write). Even for read-only access, memory barriers would be needed.
+ * In any case, it is likely that decisions based on only reading must
+ * also be atomic with respect to data held here and elsewhere in the
+ * Job Scheduler.
+ *
+ * This lock must also be held for accessing:
+ * - kbase_context::as_nr
+ * - Parts of the kbasep_js_policy, dependent on the policy (refer to
+ * the policy in question for more information)
+ * - Parts of kbasep_js_policy_ctx_info, dependent on the policy (refer to
+ * the policy in question for more information)
+ *
+ * If accessing a job slot at the same time, the slot's IRQ lock must
+ * be obtained first to respect lock ordering.
+ */
+ osk_spinlock_irq lock;
+
+ /** Bitvector indicating whether a currently scheduled context is allowed to submit jobs.
+ * When bit 'N' is set in this, it indicates whether the context bound to address space
+ * 'N' (per_as_data[N].kctx) is allowed to submit jobs.
+ *
+ * It is placed here because it's much more memory efficient than having a mali_bool8 in
+ * kbasep_js_per_as_data to store this flag */
+ u16 submit_allowed;
+
+ /** Context Attributes:
+ * Each is large enough to hold a refcount of the number of contexts
+ * that can fit into the runpool. This is currently BASE_MAX_NR_AS
+ *
+ * Note that when BASE_MAX_NR_AS==16 we need 5 bits (not 4) to store
+ * the refcount. Hence, it's not worthwhile reducing this to
+ * bit-manipulation on u32s to save space (where in contrast, 4 bit
+ * sub-fields would be easy to do and would save space).
+ *
+ * Whilst this must not become negative, the sign bit is used for:
+ * - error detection in debug builds
+ * - Optimization: it is undefined for a signed int to overflow, and so
+ * the compiler can optimize for that never happening (thus, no masking
+ * is required on updating the variable) */
+ s8 ctx_attr_ref_count[KBASEP_JS_CTX_ATTR_COUNT];
+
+ /** Data that is unique for each AS */
+ kbasep_js_per_as_data per_as_data[BASE_MAX_NR_AS];
+
+ /*
+ * Affinity management and tracking
+ */
+ /** Bitvector to aid affinity checking. Element 'n' bit 'i' indicates
+ * that slot 'n' is using core i (i.e. slot_affinity_refcount[n][i] > 0) */
+ u64 slot_affinities[BASE_JM_MAX_NR_SLOTS];
+ /** Bitvector indicating which slots \em might have atoms blocked on
+ * them because otherwise they'd violate affinity restrictions */
+ u16 slots_blocked_on_affinity;
+ /** Refcount for each core owned by each slot. Used to generate the
+ * slot_affinities array of bitvectors
+ *
+ * The value of the refcount will not exceed BASE_JM_SUBMIT_SLOTS,
+ * because it is refcounted only when a job is definitely about to be
+ * submitted to a slot, and is de-refcounted immediately after a job
+ * finishes */
+ s8 slot_affinity_refcount[BASE_JM_MAX_NR_SLOTS][64];
+
+ } runpool_irq;
+
+ /**
+ * Run Pool mutex, for managing contexts within the runpool.
+ * You must hold this lock whilst accessing any members that follow
+ *
+ * In addition, this is used to access:
+ * - the kbasep_js_kctx_info::runpool substructure
+ */
+ osk_mutex runpool_mutex;
+
+ /**
+ * Queue Lock, used to access the Policy's queue of contexts independently
+ * of the Run Pool.
+ *
+ * Of course, you don't need the Run Pool lock to access this.
+ */
+ osk_mutex queue_mutex;
+
+ u16 as_free; /**< Bitpattern of free Address Spaces */
+
+ /** Number of currently scheduled user contexts (excluding ones that are not submitting jobs) */
+ s8 nr_user_contexts_running;
+ /** Number of currently scheduled contexts (including ones that are not submitting jobs) */
+ s8 nr_all_contexts_running;
+
+ /**
+ * Policy-specific information.
+ *
+ * Refer to the structure defined by the current policy to determine which
+ * locks must be held when accessing this.
+ */
+ kbasep_js_policy policy;
+
+ /** Core Requirements to match up with base_js_atom's core_req memeber
+ * @note This is a write-once member, and so no locking is required to read */
+ base_jd_core_req js_reqs[BASE_JM_MAX_NR_SLOTS];
+
+ u32 scheduling_tick_ns; /**< Value for KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS */
+ u32 soft_stop_ticks; /**< Value for KBASE_CONFIG_ATTR_JS_SOFT_STOP_TICKS */
+ u32 hard_stop_ticks_ss; /**< Value for KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS */
+ u32 hard_stop_ticks_nss; /**< Value for KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS */
+ u32 gpu_reset_ticks_ss; /**< Value for KBASE_CONFIG_ATTR_JS_RESET_TICKS_SS */
+ u32 gpu_reset_ticks_nss; /**< Value for KBASE_CONFIG_ATTR_JS_RESET_TICKS_NSS */
+ u32 ctx_timeslice_ns; /**< Value for KBASE_CONFIG_ATTR_JS_CTX_TIMESLICE_NS */
+ u32 cfs_ctx_runtime_init_slices; /**< Value for KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_INIT_SLICES */
+ u32 cfs_ctx_runtime_min_slices; /**< Value for KBASE_CONFIG_ATTR_JS_CFS_CTX_RUNTIME_MIN_SLICES */
+#if MALI_DEBUG
+ /* Support soft-stop on a single context */
+ mali_bool softstop_always;
+#endif /* MALI_DEBUG */
+ /** The initalized-flag is placed at the end, to avoid cache-pollution (we should
+ * only be using this during init/term paths).
+ * @note This is a write-once member, and so no locking is required to read */
+ int init_status;
+} kbasep_js_device_data;
+
+
+/**
+ * @brief KBase Context Job Scheduling information structure
+ *
+ * This is a substructure in the kbase_context that encapsulates all the
+ * scheduling information.
+ */
+typedef struct kbasep_js_kctx_info
+{
+ /**
+ * Runpool substructure. This must only be accessed whilst the Run Pool
+ * mutex ( kbasep_js_device_data::runpool_mutex ) is held.
+ *
+ * In addition, the kbasep_js_device_data::runpool_irq::lock may need to be
+ * held for certain sub-members.
+ *
+ * @note some of the members could be moved into kbasep_js_device_data for
+ * improved d-cache/tlb efficiency.
+ */
+ struct
+ {
+ kbasep_js_policy_ctx_info policy_ctx; /**< Policy-specific context */
+ } runpool;
+
+ /**
+ * Job Scheduler Context information sub-structure. These members are
+ * accessed regardless of whether the context is:
+ * - In the Policy's Run Pool
+ * - In the Policy's Queue
+ * - Not queued nor in the Run Pool.
+ *
+ * You must obtain the jsctx_mutex before accessing any other members of
+ * this substructure.
+ *
+ * You may not access any of these members from IRQ context.
+ */
+ struct
+ {
+ osk_mutex jsctx_mutex; /**< Job Scheduler Context lock */
+
+ /** Number of jobs <b>ready to run</b> - does \em not include the jobs waiting in
+ * the dispatcher, and dependency-only jobs. See kbase_jd_context::job_nr
+ * for such jobs*/
+ u32 nr_jobs;
+
+ /** Context Attributes:
+ * Each is large enough to hold a refcount of the number of atoms on
+ * the context. **/
+ u32 ctx_attr_ref_count[KBASEP_JS_CTX_ATTR_COUNT];
+
+ /**
+ * Waitq that reflects whether the context is not scheduled on the run-pool.
+ * This is clear when is_scheduled is true, and set when is_scheduled
+ * is false.
+ *
+ * This waitq can be waited upon to find out when a context is no
+ * longer in the run-pool, and is used in combination with
+ * kbasep_js_policy_try_evict_ctx() to determine when it can be
+ * terminated. However, it should only be terminated once all its jobs
+ * are also terminated (see kbase_jd_context::zero_jobs_waitq).
+ *
+ * Since the waitq is only set under jsctx_mutex, the waiter should
+ * also briefly obtain and drop jsctx_mutex to guarentee that the
+ * setter has completed its work on the kbase_context.
+ */
+ osk_waitq not_scheduled_waitq;
+
+ /**
+ * Waitq that reflects whether the context is scheduled on the run-pool.
+ * This is set when is_scheduled is true, and clear when is_scheduled
+ * is false.
+ */
+ osk_waitq scheduled_waitq;
+
+ kbase_context_flags flags;
+ /* NOTE: Unify the following flags into kbase_context_flags */
+ /**
+ * Is the context scheduled on the Run Pool?
+ *
+ * This is only ever updated whilst the jsctx_mutex is held.
+ */
+ mali_bool is_scheduled;
+ mali_bool is_dying; /**< Is the context in the process of being evicted? */
+ } ctx;
+
+ /* The initalized-flag is placed at the end, to avoid cache-pollution (we should
+ * only be using this during init/term paths) */
+ int init_status;
+} kbasep_js_kctx_info;
+
+
+/**
+ * @brief The JS timer resolution, in microseconds
+ *
+ * Any non-zero difference in time will be at least this size.
+ */
+#define KBASEP_JS_TICK_RESOLUTION_US (1000000u/osk_time_mstoticks(1000))
+
+/**
+ * @note MIDBASE-769: OSK to add high resolution timer
+ *
+ * The underlying tick is an unsigned integral type
+ */
+typedef osk_ticks kbasep_js_tick;
+
+/**
+ * GPU clock ticks.
+ */
+typedef osk_ticks kbasep_js_gpu_tick;
+
+
+#endif /* _KBASE_JS_DEFS_H_ */
+
+
+/** @} */ /* end group kbase_js */
+/** @} */ /* end group base_kbase_api */
+/** @} */ /* end group base_api */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_js_policy.h
+ * Job Scheduler Policy APIs.
+ */
+
+#ifndef _KBASE_JS_POLICY_H_
+#define _KBASE_JS_POLICY_H_
+
+/**
+ * @page page_kbase_js_policy Job Scheduling Policies
+ * The Job Scheduling system is described in the following:
+ * - @subpage page_kbase_js_policy_overview
+ * - @subpage page_kbase_js_policy_operation
+ *
+ * The API details are as follows:
+ * - @ref kbase_jm
+ * - @ref kbase_js
+ * - @ref kbase_js_policy
+ */
+
+/**
+ * @page page_kbase_js_policy_overview Overview of the Policy System
+ *
+ * The Job Scheduler Policy manages:
+ * - The assigning of KBase Contexts to GPU Address Spaces (\em ASs)
+ * - The choosing of Job Chains (\em Jobs) from a KBase context, to run on the
+ * GPU's Job Slots (\em JSs).
+ * - The amount of \em time a context is assigned to (<em>scheduled on</em>) an
+ * Address Space
+ * - The amount of \em time a Job spends running on the GPU
+ *
+ * The Policy implements this management via 2 components:
+ * - A Policy Queue, which manages a set of contexts that are ready to run,
+ * but not currently running.
+ * - A Policy Run Pool, which manages the currently running contexts (one per Address
+ * Space) and the jobs to run on the Job Slots.
+ *
+ * Each Graphics Process in the system has at least one KBase Context. Therefore,
+ * the Policy Queue can be seen as a queue of Processes waiting to run Jobs on
+ * the GPU.
+ *
+ * <!-- The following needs to be all on one line, due to doxygen's parser -->
+ * @dotfile policy_overview.dot "Diagram showing a very simplified overview of the Policy System. IRQ handling, soft/hard-stopping, contexts re-entering the system and Policy details are omitted"
+ *
+ * The main operations on the queue are:
+ * - Enqueuing a Context to it
+ * - Dequeuing a Context from it, to run it.
+ * - Note: requeuing a context is much the same as enqueuing a context, but
+ * occurs when a context is scheduled out of the system to allow other contexts
+ * to run.
+ *
+ * These operations have much the same meaning for the Run Pool - Jobs are
+ * dequeued to run on a Jobslot, and requeued when they are scheduled out of
+ * the GPU.
+ *
+ * @note This is an over-simplification of the Policy APIs - there are more
+ * operations than 'Enqueue'/'Dequeue', and a Dequeue from the Policy Queue
+ * takes at least two function calls: one to Dequeue from the Queue, one to add
+ * to the Run Pool.
+ *
+ * As indicated on the diagram, Jobs permanently leave the scheduling system
+ * when they are completed, otherwise they get dequeued/requeued until this
+ * happens. Similarly, Contexts leave the scheduling system when their jobs
+ * have all completed. However, Contexts may later return to the scheduling
+ * system (not shown on the diagram) if more Bags of Jobs are submitted to
+ * them.
+ */
+
+/**
+ * @page page_kbase_js_policy_operation Policy Operation
+ *
+ * We describe the actions that the Job Scheduler Core takes on the Policy in
+ * the following cases:
+ * - The IRQ Path
+ * - The Job Submission Path
+ * - The High Priority Job Submission Path
+ *
+ * This shows how the Policy APIs will be used by the Job Scheduler core.
+ *
+ * The following diagram shows an example Policy that contains a Low Priority
+ * queue, and a Real-time (High Priority) Queue. The RT queue is examined
+ * before the LowP one on dequeuing from the head. The Low Priority Queue is
+ * ordered by time, and the RT queue is ordered by RT-priority, and then by
+ * time. In addition, it shows that the Job Scheduler Core will start a
+ * Soft-Stop Timer (SS-Timer) when it dequeue's and submits a job. The
+ * Soft-Stop time is set by a global configuration value, and must be a value
+ * appropriate for the policy. For example, this could include "don't run a
+ * soft-stop timer" for a First-Come-First-Served (FCFS) policy.
+ *
+ * <!-- The following needs to be all on one line, due to doxygen's parser -->
+ * @dotfile policy_operation_diagram.dot "Diagram showing the objects managed by an Example Policy, and the operations made upon these objects by the Job Scheduler Core."
+ *
+ * @section sec_kbase_js_policy_operation_prio Dealing with Priority
+ *
+ * Priority applies both to a context as a whole, and to the jobs within a
+ * context. The jobs specify a priority in the base_jd_atom::prio member, which
+ * is relative to that of the context. A positive setting indicates a reduction
+ * in priority, whereas a negative setting indicates a boost in priority. Of
+ * course, the boost in priority should only be honoured when the originating
+ * process has sufficient priviledges, and should be ignored for unpriviledged
+ * processes. The meaning of the combined priority value is up to the policy
+ * itself, and could be a logarithmic scale instead of a linear scale (e.g. the
+ * policy could implement an increase/decrease in priority by 1 results in an
+ * increase/decrease in \em proportion of time spent scheduled in by 25%, an
+ * effective change in timeslice by 11%).
+ *
+ * It is up to the policy whether a boost in priority boosts the priority of
+ * the entire context (e.g. to such an extent where it may pre-empt other
+ * running contexts). If it chooses to do this, the Policy must make sure that
+ * only the high-priority jobs are run, and that the context is scheduled out
+ * once only low priority jobs remain. This ensures that the low priority jobs
+ * within the context do not gain from the priority boost, yet they still get
+ * scheduled correctly with respect to other low priority contexts.
+ *
+ *
+ * @section sec_kbase_js_policy_operation_irq IRQ Path
+ *
+ * The following happens on the IRQ path from the Job Scheduler Core:
+ * - Note the slot that completed (for later)
+ * - Log the time spent by the job (and implicitly, the time spent by the
+ * context)
+ * - call kbasep_js_policy_log_job_result() <em>in the context of the irq
+ * handler.</em>
+ * - This must happen regardless of whether the job completed successfully or
+ * not (otherwise the context gets away with DoS'ing the system with faulty jobs)
+ * - What was the result of the job?
+ * - If Completed: job is just removed from the system
+ * - If Hard-stop or failure: job is removed from the system
+ * - If Soft-stop: queue the book-keeping work onto a work-queue: have a
+ * work-queue call kbasep_js_policy_enqueue_job()
+ * - Check the timeslice used by the owning context
+ * - call kbasep_js_policy_should_remove_ctx() <em>in the context of the irq
+ * handler.</em>
+ * - If this returns true, clear the "allowed" flag.
+ * - Check the ctx's flags for "allowed", "has jobs to run" and "is running
+ * jobs"
+ * - And so, should the context stay scheduled in?
+ * - If No, push onto a work-queue the work of scheduling out the old context,
+ * and getting a new one. That is:
+ * - kbasep_js_policy_runpool_remove_ctx() on old_ctx
+ * - kbasep_js_policy_enqueue_ctx() on old_ctx
+ * - kbasep_js_policy_dequeue_head_ctx() to get new_ctx
+ * - kbasep_js_policy_runpool_add_ctx() on new_ctx
+ * - (all of this work is deferred on a work-queue to keep the IRQ handler quick)
+ * - If there is space in the completed job slots' HEAD/NEXT registers, run the next job:
+ * - kbasep_js_policy_dequeue_job_irq() <em>in the context of the irq
+ * handler</em> with core_req set to that of the completing slot
+ * - if this returned MALI_TRUE, submit the job to the completed slot.
+ * - This is repeated until kbasep_js_policy_dequeue_job_irq() returns
+ * MALI_FALSE, or the job slot has a job queued on both the HEAD and NEXT registers.
+ * - If kbasep_js_policy_dequeue_job_irq() returned false, submit some work to
+ * the work-queue to retry from outside of IRQ context (calling
+ * kbasep_js_policy_dequeue_job() from a work-queue).
+ *
+ * Since the IRQ handler submits new jobs \em and re-checks the IRQ_RAWSTAT,
+ * this sequence could loop a large number of times: this could happen if
+ * the jobs submitted completed on the GPU very quickly (in a few cycles), such
+ * as GPU NULL jobs. Then, the HEAD/NEXT registers will always be free to take
+ * more jobs, causing us to loop until we run out of jobs.
+ *
+ * To mitigate this, we must limit the number of jobs submitted per slot during
+ * the IRQ handler - for example, no more than 2 jobs per slot per IRQ should
+ * be sufficient (to fill up the HEAD + NEXT registers in normal cases). For
+ * Mali-T600 with 3 job slots, this means that up to 6 jobs could be submitted per
+ * slot. Note that IRQ Throttling can make this situation commonplace: 6 jobs
+ * could complete but the IRQ for each of them is delayed by the throttling. By
+ * the time you get the IRQ, all 6 jobs could've completed, meaning you can
+ * submit jobs to fill all 6 HEAD+NEXT registers again.
+ *
+ * @note As much work is deferred as possible, which includes the scheduling
+ * out of a context and scheduling in a new context. However, we can still make
+ * starting a single high-priorty context quick despite this:
+ * - On Mali-T600 family, there is one more AS than JSs.
+ * - This means we can very quickly schedule out one AS, no matter what the
+ * situation (because there will always be one AS that's not currently running
+ * on the job slot - it can only have a job in the NEXT register).
+ * - Even with this scheduling out, fair-share can still be guaranteed e.g. by
+ * a timeline-based Completely Fair Scheduler.
+ * - When our high-priority context comes in, we can do this quick-scheduling
+ * out immediately, and then schedule in the high-priority context without having to block.
+ * - This all assumes that the context to schedule out is of lower
+ * priority. Otherwise, we will have to block waiting for some other low
+ * priority context to finish its jobs. Note that it's likely (but not
+ * impossible) that the high-priority context \b is running jobs, by virtue of
+ * it being high priority.
+ * - Therefore, we can give a high liklihood that on Mali-T600 at least one
+ * high-priority context can be started very quickly. For the general case, we
+ * can guarantee starting (no. ASs) - (no. JSs) high priority contexts
+ * quickly. In any case, there is a high likelihood that we're able to start
+ * more than one high priority context quickly.
+ *
+ * In terms of the functions used in the IRQ handler directly, these are the
+ * perfomance considerations:
+ * - kbase_js_policy_log_job_result():
+ * - This is just adding to a 64-bit value (possibly even a 32-bit value if we
+ * only store the time the job's recently spent - see below on 'priority weighting')
+ * - For priority weighting, a divide operation ('div') could happen, but
+ * this can happen in a deferred context (outside of IRQ) when scheduling out
+ * the ctx; as per our Engineering Specification, the contexts of different
+ * priority still stay scheduled in for the same timeslice, but higher priority
+ * ones scheduled back in more often.
+ * - That is, the weighted and unweighted times must be stored separately, and
+ * the weighted time is only updated \em outside of IRQ context.
+ * - Of course, this divide is more likely to be a 'multiply by inverse of the
+ * weight', assuming that the weight (priority) doesn't change.
+ * - kbasep_js_policy_should_remove_ctx():
+ * - This is usually just a comparison of the stored time value against some
+ * maximum value.
+ * - kbasep_js_policy_dequeue_job_irq():
+ * - For very fast operation, it can keep a very small buffer of 1 element per
+ * job-slot that allows the job at the head of the runpool for each job-slot
+ * to be retreived very quickly (O(1) time). This is complicated by high
+ * priority jobs that may 'jump' the queue, but could be eased by having a
+ * second buffer for high priority jobs. This assumes the requirement is only to
+ * run any high priority job quickly, not to run the highest high priority job
+ * quickly.
+ * - Of course, if a job slot completes two jobs in quick succession, then
+ * kbasep_js_policy_dequeue_job_irq() can return MALI_FALSE on the second call
+ * (because the small quick-access buffer is already exhausted)
+ * - The quick-access buffer must be refilled by the other Policy Job
+ * Management APIs that are called ourside of IRQ context.
+ * - This scheme guarantees that we keep every jobslot busy with at least one
+ * job - good utilization.
+ * - As a side effect, processes that try to submit too many quick-running
+ * jobs (to increase IRQ rate to cause a DoS attack ) will be limited to the
+ * rate at which the kernel work-queue can be serivced. This can be seen as a
+ * benefit.
+ *
+ * @note all deferred work can be wrapped up into one call - we usually need to
+ * indicate that a job/bag is done outside of IRQ context anyway.
+ *
+ *
+ *
+ * @section sec_kbase_js_policy_operation_submit Submission path
+ *
+ * Start with a Context with no jobs present, and assume equal priority of all
+ * contexts in the system. The following work all happens outside of IRQ
+ * Context :
+ * - As soon as job is made 'ready to 'run', then is must be registerd with the Job
+ * Scheduler Policy:
+ * - 'Ready to run' means they've satisified their dependencies in the
+ * Kernel-side Job Dispatch system.
+ * - Call kbasep_js_policy_enqueue_job()
+ * - This indicates that the job should be scheduled (it is ready to run).
+ * - As soon as a ctx changes from having 0 jobs 'ready to run' to >0 jobs
+ * 'ready to run', we enqueue the context on the policy queue:
+ * - Call kbasep_js_policy_enqueue_ctx()
+ * - This indicates that the \em ctx should be scheduled (it is ready to run)
+ *
+ * Next, we need to handle adding a context to the Run Pool - if it's sensible
+ * to do so. This can happen due to two reasons:
+ * -# A context is enqueued as above, and there are ASs free for it to run on
+ * (e.g. it is the first context to be run, in which case it can be added to
+ * the Run Pool immediately after enqueuing on the Policy Queue)
+ * -# A previous IRQ caused another ctx to be scheduled out, requiring that the
+ * context at the head of the queue be scheduled in. Such steps would happen in
+ * a work queue (work deferred from the IRQ context).
+ *
+ * In both cases, we'd handle it as follows:
+ * - Get the context at the Head of the Policy Queue:
+ * - Call kbasep_js_policy_dequeue_head_ctx()
+ * - Assign the Context an Address Space (Assert that there will be one free,
+ * given the above two reasons)
+ * - Add this context to the Run Pool:
+ * - Call kbasep_js_policy_runpool_add_ctx()
+ * - Now see if a job should be run:
+ * - Mostly, this will be done in the IRQ handler at the completion of a
+ * previous job.
+ * - However, there are two cases where this cannot be done: a) The first job
+ * enqueued to the system (there is no previous IRQ to act upon) b) When jobs
+ * are submitted at a low enough rate to not fill up all Job Slots (or, not to
+ * fill both the 'HEAD' and 'NEXT' registers in the job-slots)
+ * - Hence, on each ctx <b>and job</b> submission we should try to see if we
+ * can run a job:
+ * - For each job slot that has free space (in NEXT or HEAD+NEXT registers):
+ * - Call kbasep_js_policy_dequeue_job() with core_req set to that of the
+ * slot
+ * - if we got one, submit it to the job slot.
+ * - This is repeated until kbasep_js_policy_dequeue_job() returns
+ * MALI_FALSE, or the job slot has a job queued on both the HEAD and NEXT registers.
+ *
+ * The above case shows that we should attempt to run jobs in cases where a) a ctx
+ * has been added to the Run Pool, and b) new jobs have been added to a context
+ * in the Run Pool:
+ * - In the latter case, the context is in the runpool because it's got a job
+ * ready to run, or is already running a job
+ * - We could just wait until the IRQ handler fires, but for certain types of
+ * jobs this can take comparatively a long time to complete, e.g. GLES FS jobs
+ * generally take much longer to run that GLES CS jobs, which are vertex shader
+ * jobs. Even worse are NSS jobs, which may run for seconds/minutes.
+ * - Therefore, when a new job appears in the ctx, we must check the job-slots
+ * to see if they're free, and run the jobs as before.
+ *
+ *
+ *
+ * @section sec_kbase_js_policy_operation_submit_hipri Submission path for High Priority Contexts
+ *
+ * For High Priority Contexts on Mali-T600, we can make sure that at least 1 of
+ * them can be scheduled in immediately to start high prioriy jobs. In general,
+ * (no. ASs) - (no JSs) high priority contexts may be started immediately. The
+ * following describes how this happens:
+ *
+ * Similar to the previous section, consider what happens with a high-priority
+ * context (a context with a priority higher than that of any in the Run Pool)
+ * that starts out with no jobs:
+ * - A job becomes ready to run on the context, and so we enqueue the context
+ * on the Policy's Queue.
+ * - However, we'd like to schedule in this context immediately, instead of
+ * waiting for one of the Run Pool contexts' timeslice to expire
+ * - The policy's Enqueue function must detect this (because it is the policy
+ * that embodies the concept of priority), and take appropriate action
+ * - That is, kbasep_js_policy_enqueue_ctx() should check the Policy's Run
+ * Pool to see if a lower priority context should be scheduled out, and then
+ * schedule in the High Priority context.
+ * - For Mali-T600, we can always pick a context to schedule out immediately
+ * (because there are more ASs than JSs), and so scheduling out a victim context
+ * and scheduling in the high priority context can happen immediately.
+ * - If a policy implements fair-sharing, then this can still ensure the
+ * victim later on gets a fair share of the GPU.
+ * - As a note, consider whether the victim can be of equal/higher priority
+ * than the incoming context:
+ * - Usually, higher priority contexts will be the ones currently running
+ * jobs, and so the context with the lowest priority is usually not running
+ * jobs.
+ * - This makes it likely that the victim context is low priority, but
+ * it's not impossible for it to be a high priority one:
+ * - Suppose 3 high priority contexts are submitting only FS jobs, and one low
+ * priority context submitting CS jobs. Then, the context not running jobs will
+ * be one of the hi priority contexts (because only 2 FS jobs can be
+ * queued/running on the GPU HW for Mali-T600).
+ * - The problem can be mitigated by extra action, but it's questionable
+ * whether we need to: we already have a high likelihood that there's at least
+ * one high priority context - that should be good enough.
+ * - And so, this method makes sure that at least one high priority context
+ * can be started very quickly, but more than one high priority contexts could be
+ * delayed (up to one timeslice).
+ * - To improve this, use a GPU with a higher number of Address Spaces vs Job
+ * Slots.
+ * - At this point, let's assume this high priority context has been scheduled
+ * in immediately. The next step is to ensure it can start some jobs quickly.
+ * - It must do this by Soft-Stopping jobs on any of the Job Slots that it can
+ * submit to.
+ * - The rest of the logic for starting the jobs is taken care of by the IRQ
+ * handler. All the policy needs to do is ensure that
+ * kbasep_js_policy_dequeue_job() will return the jobs from the high priority
+ * context.
+ *
+ * @note in SS state, we currently only use 2 job-slots (even for T608, but
+ * this might change in future). In this case, it's always possible to schedule
+ * out 2 ASs quickly (their jobs won't be in the HEAD registers). At the same
+ * time, this maximizes usage of the job-slots (only 2 are in use), because you
+ * can guarantee starting of the jobs from the High Priority contexts immediately too.
+ *
+ *
+ *
+ * @section sec_kbase_js_policy_operation_notes Notes
+ *
+ * - In this design, a separate 'init' is needed from dequeue/requeue, so that
+ * information can be retained between the dequeue/requeue calls. For example,
+ * the total time spent for a context/job could be logged between
+ * dequeue/requeuing, to implement Fair Sharing. In this case, 'init' just
+ * initializes that information to some known state.
+ *
+ *
+ *
+ */
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_kbase_api
+ * @{
+ */
+
+/**
+ * @addtogroup kbase_js_policy Job Scheduler Policy APIs
+ * @{
+ *
+ * <b>Refer to @ref page_kbase_js_policy for an overview and detailed operation of
+ * the Job Scheduler Policy and its use from the Job Scheduler Core.</b>
+ */
+
+/**
+ * @brief Job Scheduler Policy structure
+ */
+union kbasep_js_policy;
+
+/**
+ * @brief Initialize the Job Scheduler Policy
+ */
+mali_error kbasep_js_policy_init( kbase_device *kbdev );
+
+/**
+ * @brief Terminate the Job Scheduler Policy
+ */
+void kbasep_js_policy_term( kbasep_js_policy *js_policy );
+
+
+
+/**
+ * @addtogroup kbase_js_policy_ctx Job Scheduler Policy, Context Management API
+ * @{
+ *
+ * <b>Refer to @ref page_kbase_js_policy for an overview and detailed operation of
+ * the Job Scheduler Policy and its use from the Job Scheduler Core.</b>
+ */
+
+
+/**
+ * @brief Job Scheduler Policy Ctx Info structure
+ *
+ * This structure is embedded in the kbase_context structure. It is used to:
+ * - track information needed for the policy to schedule the context (e.g. time
+ * used, OS priority etc.)
+ * - link together kbase_contexts into a queue, so that a kbase_context can be
+ * obtained as the container of the policy ctx info. This allows the API to
+ * return what "the next context" should be.
+ * - obtain other information already stored in the kbase_context for
+ * scheduling purposes (e.g process ID to get the priority of the originating
+ * process)
+ */
+union kbasep_js_policy_ctx_info;
+
+/**
+ * @brief Initialize a ctx for use with the Job Scheduler Policy
+ *
+ * This effectively initializes the kbasep_js_policy_ctx_info structure within
+ * the kbase_context (itself located within the kctx->jctx.sched_info structure).
+ */
+mali_error kbasep_js_policy_init_ctx( kbase_device *kbdev, kbase_context *kctx );
+
+/**
+ * @brief Terminate resources associated with using a ctx in the Job Scheduler
+ * Policy.
+ */
+void kbasep_js_policy_term_ctx( kbasep_js_policy *js_policy, kbase_context *kctx );
+
+/**
+ * @brief Enqueue a context onto the Job Scheduler Policy Queue
+ *
+ * If the context enqueued has a priority higher than any in the Run Pool, then
+ * it is the Policy's responsibility to decide whether to schedule out a low
+ * priority context from the Run Pool to allow the high priority context to be
+ * scheduled in.
+ *
+ * If the context has the privileged flag set, it will always be kept at the
+ * head of the queue.
+ *
+ * The caller will be holding kbasep_js_kctx_info::ctx::jsctx_mutex.
+ * The caller will be holding kbasep_js_device_data::queue_mutex.
+ */
+void kbasep_js_policy_enqueue_ctx( kbasep_js_policy *js_policy, kbase_context *kctx );
+
+/**
+ * @brief Dequeue a context from the Head of the Job Scheduler Policy Queue
+ *
+ * The caller will be holding kbasep_js_device_data::queue_mutex.
+ *
+ * @return MALI_TRUE if a context was available, and *kctx_ptr points to
+ * the kctx dequeued.
+ * @return MALI_FALSE if no contexts were available.
+ */
+mali_bool kbasep_js_policy_dequeue_head_ctx( kbasep_js_policy *js_policy, kbase_context **kctx_ptr );
+
+/**
+ * @brief Evict a context from the Job Scheduler Policy Queue
+ *
+ * This is only called as part of destroying a kbase_context.
+ *
+ * There are many reasons why this might fail during the lifetime of a
+ * context. For example, the context is in the process of being scheduled. In
+ * that case a thread doing the scheduling might have a pointer to it, but the
+ * context is neither in the Policy Queue, nor is it in the Run
+ * Pool. Crucially, neither the Policy Queue, Run Pool, or the Context itself
+ * are locked.
+ *
+ * Hence to find out where in the system the context is, it is important to do
+ * more than just check the kbasep_js_kctx_info::ctx::is_scheduled member.
+ *
+ * The caller will be holding kbasep_js_device_data::queue_mutex.
+ *
+ * @return MALI_TRUE if the context was evicted from the Policy Queue
+ * @return MALI_FALSE if the context was not found in the Policy Queue
+ */
+mali_bool kbasep_js_policy_try_evict_ctx( kbasep_js_policy *js_policy, kbase_context *kctx );
+
+/**
+ * @brief Remove all jobs belonging to a non-queued, non-running context.
+ *
+ * This must call kbase_jd_cancel() on each job belonging to the context, which
+ * causes all necessary job cleanup actions to occur on a workqueue.
+ *
+ * At the time of the call, the context is guarenteed to be not-currently
+ * scheduled on the Run Pool (is_scheduled == MALI_FALSE), and not present in
+ * the Policy Queue. This is because one of the following functions was used
+ * recently on the context:
+ * - kbasep_js_policy_evict_ctx()
+ * - kbasep_js_policy_runpool_remove_ctx()
+ *
+ * In both cases, no subsequent call was made on the context to any of:
+ * - kbasep_js_policy_runpool_add_ctx()
+ * - kbasep_js_policy_enqueue_ctx()
+ *
+ * This is only called as part of destroying a kbase_context.
+ *
+ * The locking conditions on the caller are as follows:
+ * - it will be holding kbasep_js_kctx_info::ctx::jsctx_mutex.
+ */
+void kbasep_js_policy_kill_all_ctx_jobs( kbasep_js_policy *js_policy, kbase_context *kctx );
+
+/**
+ * @brief Add a context to the Job Scheduler Policy's Run Pool
+ *
+ * If the context enqueued has a priority higher than any in the Run Pool, then
+ * it is the Policy's responsibility to decide whether to schedule out low
+ * priority jobs that are currently running on the GPU.
+ *
+ * The number of contexts present in the Run Pool will never be more than the
+ * number of Address Spaces.
+ *
+ * The following guarentees are made about the state of the system when this
+ * is called:
+ * - kctx->as_nr member is valid
+ * - the context has its submit_allowed flag set
+ * - kbasep_js_device_data::runpool_irq::per_as_data[kctx->as_nr] is valid
+ * - The refcount of the context is guarenteed to be zero.
+ * - kbasep_js_kctx_info::ctx::is_scheduled will be MALI_TRUE.
+ *
+ * The locking conditions on the caller are as follows:
+ * - it will be holding kbasep_js_kctx_info::ctx::jsctx_mutex.
+ * - it will be holding kbasep_js_device_data::runpool_mutex.
+ * - it will be holding kbasep_js_device_data::runpool_irq::lock (a spinlock)
+ *
+ * Due to a spinlock being held, this function must not call any APIs that sleep.
+ */
+void kbasep_js_policy_runpool_add_ctx( kbasep_js_policy *js_policy, kbase_context *kctx );
+
+/**
+ * @brief Remove a context from the Job Scheduler Policy's Run Pool
+ *
+ * The kctx->as_nr member is valid and the context has its submit_allowed flag
+ * set when this is called. The state of
+ * kbasep_js_device_data::runpool_irq::per_as_data[kctx->as_nr] is also
+ * valid. The refcount of the context is guarenteed to be zero.
+ *
+ * The locking conditions on the caller are as follows:
+ * - it will be holding kbasep_js_kctx_info::ctx::jsctx_mutex.
+ * - it will be holding kbasep_js_device_data::runpool_mutex.
+ * - it will be holding kbasep_js_device_data::runpool_irq::lock (a spinlock)
+ *
+ * Due to a spinlock being held, this function must not call any APIs that sleep.
+ */
+void kbasep_js_policy_runpool_remove_ctx( kbasep_js_policy *js_policy, kbase_context *kctx );
+
+/**
+ * @brief Indicate whether a context should be removed from the Run Pool
+ * (should be scheduled out).
+ *
+ * The kbasep_js_device_data::runpool_irq::lock will be held by the caller.
+ *
+ * @note This API is called from IRQ context.
+ */
+mali_bool kbasep_js_policy_should_remove_ctx( kbasep_js_policy *js_policy, kbase_context *kctx );
+
+/**
+ * @brief Indicate whether a new context has an higher priority than the current context.
+ *
+ *
+ * The caller has the following conditions on locking:
+ * - kbasep_js_kctx_info::ctx::jsctx_mutex will be held for \a new_ctx
+ *
+ * This function must not sleep, because an IRQ spinlock might be held whilst
+ * this is called.
+ *
+ * @note There is nothing to stop the priority of \a current_ctx changing
+ * during or immediately after this function is called (because its jsctx_mutex
+ * cannot be held). Therefore, this function should only be seen as a heuristic
+ * guide as to whether \a new_ctx is higher priority than \a current_ctx
+ */
+mali_bool kbasep_js_policy_ctx_has_priority( kbasep_js_policy *js_policy, kbase_context *current_ctx, kbase_context *new_ctx );
+
+
+/** @} */ /* end group kbase_js_policy_ctx */
+
+/**
+ * @addtogroup kbase_js_policy_job Job Scheduler Policy, Job Chain Management API
+ * @{
+ *
+ * <b>Refer to @ref page_kbase_js_policy for an overview and detailed operation of
+ * the Job Scheduler Policy and its use from the Job Scheduler Core.</b>
+ */
+
+
+
+/**
+ * @brief Job Scheduler Policy Job Info structure
+ *
+ * This structure is embedded in the kbase_jd_atom structure. It is used to:
+ * - track information needed for the policy to schedule the job (e.g. time
+ * used, OS priority etc.)
+ * - link together jobs into a queue/buffer, so that a kbase_jd_atom can be
+ * obtained as the container of the policy job info. This allows the API to
+ * return what "the next job" should be.
+ * - obtain other information already stored in the kbase_context for
+ * scheduling purposes (e.g user-side relative priority)
+ */
+union kbasep_js_policy_job_info;
+
+/**
+ * @brief Initialize a job for use with the Job Scheduler Policy
+ *
+ * This function initializes the kbasep_js_policy_job_info structure within the
+ * kbase_jd_atom. It will only initialize/allocate resources that are specific
+ * to the job.
+ *
+ * That is, this function makes \b no attempt to:
+ * - initialize any context/policy-wide information
+ * - enqueue the job on the policy.
+ *
+ * At some later point, the following functions must be called on the job, in this order:
+ * - kbasep_js_policy_register_job() to register the job and initialize policy/context wide data.
+ * - kbasep_js_policy_enqueue_job() to enqueue the job
+ *
+ * A job must only ever be initialized on the Policy once, and must be
+ * terminated on the Policy before the job is freed.
+ *
+ * The caller will not be holding any locks, and so this function will not
+ * modify any information in \a kctx or \a js_policy.
+ *
+ * @return MALI_ERROR_NONE if initialization was correct.
+ */
+mali_error kbasep_js_policy_init_job( const kbasep_js_policy *js_policy, const kbase_context *kctx, kbase_jd_atom *katom );
+
+/**
+ * @brief Terminate resources associated with using a job in the Job Scheduler
+ * Policy.
+ *
+ * kbasep_js_policy_deregister_job() must have been called on \a katom before
+ * calling this.
+ *
+ * The caller will not be holding any locks, and so this function will not
+ * modify any information in \a kctx or \a js_policy.
+ */
+void kbasep_js_policy_term_job( const kbasep_js_policy *js_policy, const kbase_context *kctx, kbase_jd_atom *katom );
+
+/**
+ * @brief Register context/policy-wide information for a job on the Job Scheduler Policy.
+ *
+ * Registers the job with the policy. This is used to track the job before it
+ * has been enqueued/requeued by kbasep_js_policy_enqueue_job(). Specifically,
+ * it is used to update information under a lock that could not be updated at
+ * kbasep_js_policy_init_job() time (such as context/policy-wide data).
+ *
+ * @note This function will not fail, and hence does not allocate any
+ * resources. Any failures that could occur on registration will be caught
+ * during kbasep_js_policy_init_job() instead.
+ *
+ * A job must only ever be registerd on the Policy once, and must be
+ * deregistered on the Policy on completion (whether or not that completion was
+ * success/failure).
+ *
+ * The caller has the following conditions on locking:
+ * - kbasep_js_kctx_info::ctx::jsctx_mutex will be held.
+ */
+void kbasep_js_policy_register_job( kbasep_js_policy *js_policy, kbase_context *kctx, kbase_jd_atom *katom );
+
+/**
+ * @brief De-register context/policy-wide information for a on the Job Scheduler Policy.
+ *
+ * This must be used before terminating the resources associated with using a
+ * job in the Job Scheduler Policy. That is, it must be called before
+ * kbasep_js_policy_term_job(). This function does not itself terminate any
+ * resources, at most it just updates information in the policy and context.
+ *
+ * The caller has the following conditions on locking:
+ * - kbasep_js_kctx_info::ctx::jsctx_mutex will be held.
+ */
+void kbasep_js_policy_deregister_job( kbasep_js_policy *js_policy, kbase_context *kctx, kbase_jd_atom *katom );
+
+
+/**
+ * @brief Dequeue a Job for a job slot from the Job Scheduler Policy Run Pool
+ *
+ * The job returned by the policy will match at least one of the bits in the
+ * job slot's core requirements (but it may match more than one, or all @ref
+ * base_jd_core_req bits supported by the job slot).
+ *
+ * In addition, the requirements of the job returned will be a subset of those
+ * requested - the job returned will not have requirements that \a job_slot_idx
+ * cannot satisfy.
+ *
+ * The caller will submit the job to the GPU as soon as the GPU's NEXT register
+ * for the corresponding slot is empty. Of course, the GPU will then only run
+ * this new job when the currently executing job (in the jobslot's HEAD
+ * register) has completed.
+ *
+ * @return MALI_TRUE if a job was available, and *kctx_ptr points to
+ * the kctx dequeued.
+ * @return MALI_FALSE if no jobs were available among all ctxs in the Run Pool.
+ *
+ * @note base_jd_core_req is currently a u8 - beware of type conversion.
+ *
+ * @note This API is not called from IRQ context outside of the policy
+ * itself, and so need not operate in O(1) time. Refer to
+ * kbasep_js_policy_dequeue_job_irq() for dequeuing from IRQ context.
+ *
+ * As a result of kbasep_js_policy_dequeue_job_irq(), this function might need to
+ * carry out work to maintain its internal queues both before and after a job
+ * is dequeued.
+ *
+ * The caller has the following conditions on locking:
+ * - kbasep_js_device_data::runpool_lock::irq will be held.
+ * - kbdev->jm_slots[ job_slot_idx ].lock will be held
+ * - kbasep_js_device_data::runpool_mutex will be held.
+ * - kbasep_js_kctx_info::ctx::jsctx_mutex. will be held
+ */
+mali_bool kbasep_js_policy_dequeue_job( kbase_device *kbdev,
+ int job_slot_idx,
+ kbase_jd_atom **katom_ptr );
+
+/**
+ * @brief IRQ Context Fast equivalent of kbasep_js_policy_dequeue_job()
+ *
+ * This is a 'fast' variant of kbasep_js_policy_dequeue_job() that will be
+ * called from IRQ context.
+ *
+ * It is recommended that this is coded to be O(1) and must be capable of
+ * returning at least one job per job-slot to IRQ context. If IRQs occur in
+ * quick succession without any work done in non-irq context, then this
+ * function is allowed to return MALI_FALSE even if there are jobs available
+ * that satisfy the requirements.
+ *
+ * This relaxation of correct dequeuing allows O(1) execution with bounded
+ * memory requirements. For example, in addition to the ctxs' job queues the run
+ * pool can have a buffer that can contain a single job for 'quick access' per job
+ * slot, but this buffer is only refilled from the job queue outside of IRQ
+ * context.
+ *
+ * Therefore, all other Job Scheduled Policy Job Management APIs can be
+ * implemented to refill this buffer/maintain the Run Pool's job queues outside
+ * of IRQ context.
+ *
+ * The caller has the following conditions on locking:
+ * - kbasep_js_device_data::runpool_irq::lock will be held.
+ * - kbdev->jm_slots[ job_slot_idx ].lock will be held
+ *
+ * @note The caller \em might be holding one of the
+ * kbasep_js_kctx_info::ctx::jsctx_mutex locks, if this code is called from
+ * outside of IRQ context.
+ */
+mali_bool kbasep_js_policy_dequeue_job_irq( kbase_device *kbdev,
+ int job_slot_idx,
+ kbase_jd_atom **katom_ptr );
+
+
+/**
+ * @brief Requeue a Job back into the the Job Scheduler Policy Run Pool
+ *
+ * This will be used to enqueue a job after its creation and also to requeue
+ * a job into the Run Pool that was previously dequeued (running). It notifies
+ * the policy that the job should be run again at some point later.
+ *
+ * As a result of kbasep_js_policy_dequeue_job_irq(), this function might need to
+ * carry out work to maintain its internal queues both before and after a job
+ * is requeued.
+ *
+ * The caller has the following conditions on locking:
+ * - kbasep_js_device_data::runpool_irq::lock (a spinlock) will be held.
+ * - kbasep_js_device_data::runpool_mutex will be held.
+ * - kbasep_js_kctx_info::ctx::jsctx_mutex will be held.
+ */
+void kbasep_js_policy_enqueue_job( kbasep_js_policy *js_policy, kbase_jd_atom *katom );
+
+
+/**
+ * @brief Log the result of a job: the time spent on a job/context, and whether
+ * the job failed or not.
+ *
+ * Since a kbase_jd_atom contains a pointer to the kbase_context owning it,
+ * then this can also be used to log time on either/both the job and the
+ * containing context.
+ *
+ * The completion state of the job can be found by examining \a katom->event.event_code
+ *
+ * If the Job failed and the policy is implementing fair-sharing, then the
+ * policy must penalize the failing job/context:
+ * - At the very least, it should penalize the time taken by the amount of
+ * time spent processing the IRQ in SW. This because a job in the NEXT slot
+ * waiting to run will be delayed until the failing job has had the IRQ
+ * cleared.
+ * - \b Optionally, the policy could apply other penalties. For example, based
+ * on a threshold of a number of failing jobs, after which a large penalty is
+ * applied.
+ *
+ * The kbasep_js_device_data::runpool_mutex will be held by the caller.
+ *
+ * @note This API is called from IRQ context.
+ *
+ * The caller has the following conditions on locking:
+ * - kbasep_js_device_data::runpool_irq::lock will be held.
+ *
+ * @param js_policy job scheduler policy
+ * @param katom job dispatch atom
+ * @param time_spent_us the time spent by the job, in microseconds (10^-6 seconds).
+ */
+void kbasep_js_policy_log_job_result( kbasep_js_policy *js_policy, kbase_jd_atom *katom, u32 time_spent_us );
+
+/** @} */ /* end group kbase_js_policy_job */
+
+
+
+/** @} */ /* end group kbase_js_policy */
+/** @} */ /* end group base_kbase_api */
+/** @} */ /* end group base_api */
+
+#endif /* _KBASE_JS_POLICY_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/*
+ * Job Scheduler: Completely Fair Policy Implementation
+ */
+
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_kbase_jm.h>
+#include <kbase/src/common/mali_kbase_js.h>
+#include <kbase/src/common/mali_kbase_js_policy_cfs.h>
+
+/**
+ * Define for when dumping is enabled.
+ * This should not be based on the instrumentation level as whether dumping is enabled for a particular level is down to the integrator.
+ * However this is being used for now as otherwise the cinstr headers would be needed.
+ */
+#define CINSTR_DUMPING_ENABLED ( 2 == MALI_INSTRUMENTATION_LEVEL )
+
+/** Fixed point constants used for runtime weight calculations */
+#define WEIGHT_FIXEDPOINT_SHIFT 10
+#define WEIGHT_TABLE_SIZE 40
+#define WEIGHT_0_NICE (WEIGHT_TABLE_SIZE/2)
+#define WEIGHT_0_VAL (1 << WEIGHT_FIXEDPOINT_SHIFT)
+
+#define LOOKUP_VARIANT_MASK ((1u<<KBASEP_JS_MAX_NR_CORE_REQ_VARIANTS) - 1u)
+
+/** Core requirements that all the variants support */
+#define JS_CORE_REQ_ALL_OTHERS \
+ ( BASE_JD_REQ_CF | BASE_JD_REQ_V | BASE_JD_REQ_PERMON | BASE_JD_REQ_EXTERNAL_RESOURCES )
+
+/** Context requirements the all the variants support */
+
+/* In HW issue 8987 workaround, restrict Compute-only contexts and Compute jobs onto job slot[2],
+ * which will ensure their affinity does not intersect GLES jobs */
+#define JS_CTX_REQ_ALL_OTHERS_8987 \
+ ( KBASE_CTX_FLAG_CREATE_FLAGS_SET | KBASE_CTX_FLAG_PRIVILEGED )
+#define JS_CORE_REQ_COMPUTE_SLOT_8987 \
+ ( BASE_JD_REQ_CS )
+#define JS_CORE_REQ_ONLY_COMPUTE_SLOT_8987 \
+ ( BASE_JD_REQ_ONLY_COMPUTE )
+
+/* Otherwise, compute-only contexts/compute jobs can use any job slot */
+#define JS_CTX_REQ_ALL_OTHERS \
+ ( KBASE_CTX_FLAG_CREATE_FLAGS_SET | KBASE_CTX_FLAG_PRIVILEGED | KBASE_CTX_FLAG_HINT_ONLY_COMPUTE)
+#define JS_CORE_REQ_COMPUTE_SLOT \
+ ( BASE_JD_REQ_CS | BASE_JD_REQ_ONLY_COMPUTE )
+
+/* core_req variants are ordered by least restrictive first, so that our
+ * algorithm in cached_variant_idx_init picks the least restrictive variant for
+ * each job . Note that coherent_group requirement is added to all CS variants as the
+ * selection of job-slot does not depend on the coherency requirement. */
+static const kbasep_atom_req core_req_variants[] ={
+ {
+ /* 0: Fragment variant */
+ (JS_CORE_REQ_ALL_OTHERS | BASE_JD_REQ_FS | BASE_JD_REQ_COHERENT_GROUP),
+ (JS_CTX_REQ_ALL_OTHERS),
+ 0
+ },
+ {
+ /* 1: Compute variant, can use all coregroups */
+ (JS_CORE_REQ_ALL_OTHERS | JS_CORE_REQ_COMPUTE_SLOT),
+ (JS_CTX_REQ_ALL_OTHERS),
+ 0
+ },
+ {
+ /* 2: Compute variant, uses only coherent coregroups */
+ (JS_CORE_REQ_ALL_OTHERS | JS_CORE_REQ_COMPUTE_SLOT | BASE_JD_REQ_COHERENT_GROUP ),
+ (JS_CTX_REQ_ALL_OTHERS),
+ 0
+ },
+ {
+ /* 3: Compute variant, might only use coherent coregroup, and must use tiling */
+ (JS_CORE_REQ_ALL_OTHERS | JS_CORE_REQ_COMPUTE_SLOT | BASE_JD_REQ_COHERENT_GROUP | BASE_JD_REQ_T),
+ (JS_CTX_REQ_ALL_OTHERS),
+ 0
+ },
+
+ {
+ /* 4: Variant guarenteed to support NSS atoms.
+ *
+ * In the case of a context that's specified as 'Only Compute', it'll
+ * not allow Tiler or Fragment atoms, and so those get rejected */
+ (JS_CORE_REQ_ALL_OTHERS | JS_CORE_REQ_COMPUTE_SLOT | BASE_JD_REQ_COHERENT_GROUP | BASE_JD_REQ_NSS ),
+ (JS_CTX_REQ_ALL_OTHERS),
+ 0
+ },
+
+ {
+ /* 5: Compute variant for specific-coherent-group targetting CoreGroup 0 */
+ (JS_CORE_REQ_ALL_OTHERS | JS_CORE_REQ_COMPUTE_SLOT | BASE_JD_REQ_COHERENT_GROUP | BASE_JD_REQ_SPECIFIC_COHERENT_GROUP ),
+ (JS_CTX_REQ_ALL_OTHERS),
+ 0 /* device_nr */
+ },
+ {
+ /* 6: Compute variant for specific-coherent-group targetting CoreGroup 1 */
+ (JS_CORE_REQ_ALL_OTHERS | JS_CORE_REQ_COMPUTE_SLOT | BASE_JD_REQ_COHERENT_GROUP | BASE_JD_REQ_SPECIFIC_COHERENT_GROUP ),
+ (JS_CTX_REQ_ALL_OTHERS),
+ 1 /* device_nr */
+ },
+
+ /* Unused core_req variants, to bring the total up to a power of 2 */
+ {
+ /* 7 */
+ 0,
+ 0,
+ 0
+ },
+};
+
+static const kbasep_atom_req core_req_variants_8987[] ={
+ {
+ /* 0: Fragment variant */
+ (JS_CORE_REQ_ALL_OTHERS | BASE_JD_REQ_FS | BASE_JD_REQ_COHERENT_GROUP),
+ (JS_CTX_REQ_ALL_OTHERS_8987),
+ 0
+ },
+ {
+ /* 1: Compute variant, can use all coregroups */
+ (JS_CORE_REQ_ALL_OTHERS | JS_CORE_REQ_COMPUTE_SLOT_8987),
+ (JS_CTX_REQ_ALL_OTHERS_8987),
+ 0
+ },
+ {
+ /* 2: Compute variant, uses only coherent coregroups */
+ (JS_CORE_REQ_ALL_OTHERS | JS_CORE_REQ_COMPUTE_SLOT_8987 | BASE_JD_REQ_COHERENT_GROUP ),
+ (JS_CTX_REQ_ALL_OTHERS_8987),
+ 0
+ },
+ {
+ /* 3: Compute variant, might only use coherent coregroup, and must use tiling */
+ (JS_CORE_REQ_ALL_OTHERS | JS_CORE_REQ_COMPUTE_SLOT_8987 | BASE_JD_REQ_COHERENT_GROUP | BASE_JD_REQ_T),
+ (JS_CTX_REQ_ALL_OTHERS_8987),
+ 0
+ },
+
+ {
+ /* 4: Variant guarenteed to support Compute contexts/atoms
+ *
+ * In the case of a context that's specified as 'Only Compute', it'll
+ * not allow Tiler or Fragment atoms, and so those get rejected
+ *
+ * NOTE: NSS flag cannot be supported, so its flag is cleared on bag
+ * submit */
+ (JS_CORE_REQ_ALL_OTHERS | JS_CORE_REQ_ONLY_COMPUTE_SLOT_8987 | BASE_JD_REQ_COHERENT_GROUP ),
+ (JS_CTX_REQ_ALL_OTHERS_8987 | KBASE_CTX_FLAG_HINT_ONLY_COMPUTE),
+ 0
+ },
+
+ {
+ /* 5: Compute variant for specific-coherent-group targetting CoreGroup 0
+ * Specifically, this only allows 'Only Compute' contexts/atoms */
+ (JS_CORE_REQ_ALL_OTHERS | JS_CORE_REQ_ONLY_COMPUTE_SLOT_8987 | BASE_JD_REQ_COHERENT_GROUP | BASE_JD_REQ_SPECIFIC_COHERENT_GROUP ),
+ (JS_CTX_REQ_ALL_OTHERS_8987 | KBASE_CTX_FLAG_HINT_ONLY_COMPUTE),
+ 0 /* device_nr */
+ },
+ {
+ /* 6: Compute variant for specific-coherent-group targetting CoreGroup 1
+ * Specifically, this only allows 'Only Compute' contexts/atoms */
+ (JS_CORE_REQ_ALL_OTHERS | JS_CORE_REQ_ONLY_COMPUTE_SLOT_8987 | BASE_JD_REQ_COHERENT_GROUP | BASE_JD_REQ_SPECIFIC_COHERENT_GROUP ),
+ (JS_CTX_REQ_ALL_OTHERS_8987 | KBASE_CTX_FLAG_HINT_ONLY_COMPUTE),
+ 1 /* device_nr */
+ },
+ /* Unused core_req variants, to bring the total up to a power of 2 */
+ {
+ /* 7 */
+ 0,
+ 0,
+ 0
+ },
+};
+
+#define CORE_REQ_VARIANT_FRAGMENT 0
+#define CORE_REQ_VARIANT_COMPUTE_ALL_CORES 1
+#define CORE_REQ_VARIANT_COMPUTE_ONLY_COHERENT_GROUP 2
+#define CORE_REQ_VARIANT_COMPUTE_OR_TILING 3
+#define CORE_REQ_VARIANT_COMPUTE_NSS 4
+#define CORE_REQ_VARIANT_COMPUTE_SPECIFIC_COHERENT_0 5
+#define CORE_REQ_VARIANT_COMPUTE_SPECIFIC_COHERENT_1 6
+
+#define CORE_REQ_VARIANT_ONLY_COMPUTE_8987 4
+#define CORE_REQ_VARIANT_ONLY_COMPUTE_8987_SPECIFIC_COHERENT_0 5
+#define CORE_REQ_VARIANT_ONLY_COMPUTE_8987_SPECIFIC_COHERENT_1 6
+
+
+#define NUM_CORE_REQ_VARIANTS NELEMS(core_req_variants)
+#define NUM_CORE_REQ_VARIANTS_8987 NELEMS(core_req_variants_8987)
+
+/** Mappings between job slot and variant lists for Soft-Stoppable State */
+static const u32 variants_supported_ss_state[] =
+{
+ /* js[0] uses Fragment only */
+ (1u << CORE_REQ_VARIANT_FRAGMENT),
+
+ /* js[1] uses: Compute-all-cores, Compute-only-coherent, Compute-or-Tiling,
+ * compute-specific-coregroup-0 */
+ (1u << CORE_REQ_VARIANT_COMPUTE_ALL_CORES)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_ONLY_COHERENT_GROUP)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_OR_TILING)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_SPECIFIC_COHERENT_0),
+
+ /* js[2] uses: Compute-only-coherent, compute-specific-coregroup-1 */
+ (1u << CORE_REQ_VARIANT_COMPUTE_ONLY_COHERENT_GROUP)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_SPECIFIC_COHERENT_1)
+};
+
+/** Mappings between job slot and variant lists for Soft-Stoppable State, when
+ * we have atoms that can use all the cores (KBASEP_JS_CTX_ATTR_COMPUTE_ALL_CORES)
+ * and there's more than one coregroup */
+static const u32 variants_supported_ss_allcore_state[] =
+{
+ /* js[0] uses Fragment only */
+ (1u << CORE_REQ_VARIANT_FRAGMENT),
+
+ /* js[1] uses: Compute-all-cores, Compute-only-coherent, Compute-or-Tiling,
+ * compute-specific-coregroup-0, compute-specific-coregroup-1 */
+ (1u << CORE_REQ_VARIANT_COMPUTE_ALL_CORES)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_ONLY_COHERENT_GROUP)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_OR_TILING)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_SPECIFIC_COHERENT_0)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_SPECIFIC_COHERENT_1),
+
+ /* js[2] not used */
+ 0
+};
+
+/** Mappings between job slot and variant lists for Soft-Stoppable State for
+ * BASE_HW_ISSUE_8987
+ *
+ * @note There is no 'allcores' variant of this, because this HW issue forces all
+ * atoms with BASE_JD_CORE_REQ_SPECIFIC_COHERENT_GROUP to use slot 2 anyway -
+ * hence regardless of whether a specific coregroup is targetted, those atoms
+ * still make progress. */
+static const u32 variants_supported_ss_state_8987[] =
+{
+ /* js[0] uses Fragment only */
+ (1u << CORE_REQ_VARIANT_FRAGMENT),
+
+ /* js[1] uses: Compute-all-cores, Compute-only-coherent, Compute-or-Tiling*/
+ (1u << CORE_REQ_VARIANT_COMPUTE_ALL_CORES)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_ONLY_COHERENT_GROUP)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_OR_TILING),
+
+ /* js[2] uses: All Only-compute atoms (including those targetting a
+ * specific coregroup), and nothing else. This is because their affinity
+ * must not intersect with non-only-compute atoms.
+ *
+ * As a side effect, this causes the 'device_nr' for atoms targetting a
+ * specific coregroup to be ignored */
+ (1u << CORE_REQ_VARIANT_ONLY_COMPUTE_8987)
+ | (1u << CORE_REQ_VARIANT_ONLY_COMPUTE_8987_SPECIFIC_COHERENT_0)
+ | (1u << CORE_REQ_VARIANT_ONLY_COMPUTE_8987_SPECIFIC_COHERENT_1)
+};
+
+/** Mappings between job slot and variant lists for Non-Soft-Stoppable State
+ *
+ * @note There is no 'allcores' variant of this, because NSS state forces all
+ * atoms with BASE_JD_CORE_REQ_SPECIFIC_COHERENT_GROUP to use slot 1 anyway -
+ * hence regardless of whether a specific coregroup is targetted, those atoms
+ * still make progress.
+ *
+ * @note This is effectively not used during BASE_HW_ISSUE_8987, because the
+ * NSS flag is cleared from all atoms */
+static const u32 variants_supported_nss_state[] =
+{
+ /* js[0] uses Fragment only */
+ (1u << CORE_REQ_VARIANT_FRAGMENT),
+
+ /* js[1] uses: Compute-all-cores, Compute-only-coherent, Compute-or-Tiling,
+ * Compute-targetting-specific-coregroup
+ *
+ * Due to NSS atoms, this causes the 'device_nr' for atoms targetting a
+ * specific coregroup to be ignored (otherwise the Non-NSS atoms targetting
+ * a coregroup would be unreasonably delayed) */
+ (1u << CORE_REQ_VARIANT_COMPUTE_ALL_CORES)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_ONLY_COHERENT_GROUP)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_OR_TILING)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_SPECIFIC_COHERENT_0)
+ | (1u << CORE_REQ_VARIANT_COMPUTE_SPECIFIC_COHERENT_1),
+
+ /* js[2] uses: NSS only */
+ (1u << CORE_REQ_VARIANT_COMPUTE_NSS)
+};
+
+/* Defines for easy asserts 'is scheduled'/'is queued'/'is neither queued norscheduled' */
+#define KBASEP_JS_CHECKFLAG_QUEUED (1u << 0) /**< Check the queued state */
+#define KBASEP_JS_CHECKFLAG_SCHEDULED (1u << 1) /**< Check the scheduled state */
+#define KBASEP_JS_CHECKFLAG_IS_QUEUED (1u << 2) /**< Expect queued state to be set */
+#define KBASEP_JS_CHECKFLAG_IS_SCHEDULED (1u << 3) /**< Expect scheduled state to be set */
+
+enum
+{
+ KBASEP_JS_CHECK_NOTQUEUED = KBASEP_JS_CHECKFLAG_QUEUED,
+ KBASEP_JS_CHECK_NOTSCHEDULED = KBASEP_JS_CHECKFLAG_SCHEDULED,
+ KBASEP_JS_CHECK_QUEUED = KBASEP_JS_CHECKFLAG_QUEUED | KBASEP_JS_CHECKFLAG_IS_QUEUED,
+ KBASEP_JS_CHECK_SCHEDULED = KBASEP_JS_CHECKFLAG_SCHEDULED | KBASEP_JS_CHECKFLAG_IS_SCHEDULED
+};
+
+typedef u32 kbasep_js_check;
+
+/*
+ * Private Functions
+ */
+
+/* Table autogenerated using util built from: kbase/scripts/gen_cfs_weight_of_prio.c */
+
+/* weight = 1.25 */
+static const int weight_of_priority[] =
+{
+ /* -20 */ 11, 14, 18, 23,
+ /* -16 */ 29, 36, 45, 56,
+ /* -12 */ 70, 88, 110, 137,
+ /* -8 */ 171, 214, 268, 335,
+ /* -4 */ 419, 524, 655, 819,
+ /* 0 */ 1024, 1280, 1600, 2000,
+ /* 4 */ 2500, 3125, 3906, 4883,
+ /* 8 */ 6104, 7630, 9538, 11923,
+ /* 12 */ 14904, 18630, 23288, 29110,
+ /* 16 */ 36388, 45485, 56856, 71070
+};
+
+/**
+ * @note There is nothing to stop the priority of the ctx containing \a
+ * ctx_info changing during or immediately after this function is called
+ * (because its jsctx_mutex cannot be held during IRQ). Therefore, this
+ * function should only be seen as a heuristic guide as to the priority weight
+ * of the context.
+ */
+STATIC u64 priority_weight(kbasep_js_policy_cfs_ctx *ctx_info, u32 time_us)
+{
+ u64 time_delta_us;
+ int priority;
+ priority = ctx_info->process_priority + ctx_info->bag_priority;
+
+ /* Adjust runtime_us using priority weight if required */
+ if(priority != 0 && time_us != 0)
+ {
+ int clamped_priority;
+
+ /* Clamp values to min..max weights */
+ if(priority > OSK_PROCESS_PRIORITY_MAX)
+ {
+ clamped_priority = OSK_PROCESS_PRIORITY_MAX;
+ }
+ else if(priority < OSK_PROCESS_PRIORITY_MIN)
+ {
+ clamped_priority = OSK_PROCESS_PRIORITY_MIN;
+ }
+ else
+ {
+ clamped_priority = priority;
+ }
+
+ /* Fixed point multiplication */
+ time_delta_us = ((u64)time_us * weight_of_priority[WEIGHT_0_NICE + clamped_priority]);
+ /* Remove fraction */
+ time_delta_us = time_delta_us >> WEIGHT_FIXEDPOINT_SHIFT;
+ /* Make sure the time always increases */
+ if(0 == time_delta_us)
+ {
+ time_delta_us++;
+ }
+ }
+ else
+ {
+ time_delta_us = time_us;
+ }
+
+ return time_delta_us;
+}
+
+#if KBASE_TRACE_ENABLE != 0
+STATIC int kbasep_js_policy_trace_get_refcnt_nolock( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ int as_nr;
+ int refcnt = 0;
+
+ js_devdata = &kbdev->js_data;
+
+ as_nr = kctx->as_nr;
+ if ( as_nr != KBASEP_AS_NR_INVALID )
+ {
+ kbasep_js_per_as_data *js_per_as_data;
+ js_per_as_data = &js_devdata->runpool_irq.per_as_data[as_nr];
+
+ refcnt = js_per_as_data->as_busy_refcount;
+ }
+
+ return refcnt;
+}
+
+STATIC INLINE int kbasep_js_policy_trace_get_refcnt( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ int refcnt = 0;
+
+ js_devdata = &kbdev->js_data;
+
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+ refcnt = kbasep_js_policy_trace_get_refcnt_nolock( kbdev, kctx );
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+
+ return refcnt;
+}
+#else /* KBASE_TRACE_ENABLE != 0 */
+STATIC int kbasep_js_policy_trace_get_refcnt_nolock( kbase_device *kbdev, kbase_context *kctx )
+{
+ CSTD_UNUSED( kbdev );
+ CSTD_UNUSED( kctx );
+ return 0;
+}
+
+STATIC INLINE int kbasep_js_policy_trace_get_refcnt( kbase_device *kbdev, kbase_context *kctx )
+{
+ CSTD_UNUSED( kbdev );
+ CSTD_UNUSED( kctx );
+ return 0;
+}
+#endif /* KBASE_TRACE_ENABLE != 0 */
+
+
+#if MALI_DEBUG != 0
+STATIC void kbasep_js_debug_check( kbasep_js_policy_cfs *policy_info, kbase_context *kctx, kbasep_js_check check_flag )
+{
+ /* This function uses the ternary operator and non-explicit comparisons,
+ * because it makes for much shorter, easier to read code */
+
+ if ( check_flag & KBASEP_JS_CHECKFLAG_QUEUED )
+ {
+ mali_bool is_queued;
+ mali_bool expect_queued;
+ is_queued = ( OSK_DLIST_MEMBER_OF( &policy_info->ctx_queue_head,
+ kctx,
+ jctx.sched_info.runpool.policy_ctx.cfs.list ) )? MALI_TRUE: MALI_FALSE;
+
+ if(!is_queued)
+ {
+ is_queued = ( OSK_DLIST_MEMBER_OF( &policy_info->ctx_rt_queue_head,
+ kctx,
+ jctx.sched_info.runpool.policy_ctx.cfs.list ) )? MALI_TRUE: MALI_FALSE;
+ }
+
+ expect_queued = ( check_flag & KBASEP_JS_CHECKFLAG_IS_QUEUED ) ? MALI_TRUE : MALI_FALSE;
+
+ OSK_ASSERT_MSG( expect_queued == is_queued,
+ "Expected context %p to be %s but it was %s\n",
+ kctx,
+ (expect_queued) ?"queued":"not queued",
+ (is_queued) ?"queued":"not queued" );
+
+ }
+
+ if ( check_flag & KBASEP_JS_CHECKFLAG_SCHEDULED )
+ {
+ mali_bool is_scheduled;
+ mali_bool expect_scheduled;
+ is_scheduled = ( OSK_DLIST_MEMBER_OF( &policy_info->scheduled_ctxs_head,
+ kctx,
+ jctx.sched_info.runpool.policy_ctx.cfs.list ) )? MALI_TRUE: MALI_FALSE;
+
+ expect_scheduled = ( check_flag & KBASEP_JS_CHECKFLAG_IS_SCHEDULED ) ? MALI_TRUE : MALI_FALSE;
+ OSK_ASSERT_MSG( expect_scheduled == is_scheduled,
+ "Expected context %p to be %s but it was %s\n",
+ kctx,
+ (expect_scheduled)?"scheduled":"not scheduled",
+ (is_scheduled) ?"scheduled":"not scheduled" );
+
+ }
+
+}
+#else /* MALI_DEBUG != 0 */
+STATIC void kbasep_js_debug_check( kbasep_js_policy_cfs *policy_info, kbase_context *kctx, kbasep_js_check check_flag )
+{
+ CSTD_UNUSED( policy_info );
+ CSTD_UNUSED( kctx );
+ CSTD_UNUSED( check_flag );
+ return;
+}
+#endif /* MALI_DEBUG != 0 */
+
+STATIC INLINE void set_slot_to_variant_lookup( u32 *bit_array, u32 slot_idx, u32 variants_supported )
+{
+ u32 overall_bit_idx = slot_idx * KBASEP_JS_MAX_NR_CORE_REQ_VARIANTS;
+ u32 word_idx = overall_bit_idx / 32;
+ u32 bit_idx = overall_bit_idx % 32;
+
+ OSK_ASSERT( slot_idx < BASE_JM_MAX_NR_SLOTS );
+ OSK_ASSERT( (variants_supported & ~LOOKUP_VARIANT_MASK) == 0 );
+
+ bit_array[word_idx] |= variants_supported << bit_idx;
+}
+
+
+STATIC INLINE u32 get_slot_to_variant_lookup( u32 *bit_array, u32 slot_idx )
+{
+ u32 overall_bit_idx = slot_idx * KBASEP_JS_MAX_NR_CORE_REQ_VARIANTS;
+ u32 word_idx = overall_bit_idx / 32;
+ u32 bit_idx = overall_bit_idx % 32;
+
+ u32 res;
+
+ OSK_ASSERT( slot_idx < BASE_JM_MAX_NR_SLOTS );
+
+ res = bit_array[word_idx] >> bit_idx;
+ res &= LOOKUP_VARIANT_MASK;
+
+ return res;
+}
+
+/* Check the core_req_variants: make sure that every job slot is satisifed by
+ * one of the variants. This checks that cached_variant_idx_init will produce a
+ * valid result for jobs that make maximum use of the job slots.
+ *
+ * @note The checks are limited to the job slots - this does not check that
+ * every context requirement is covered (because some are intentionally not
+ * supported, such as KBASE_CTX_FLAG_SUBMIT_DISABLED) */
+#if MALI_DEBUG
+STATIC void debug_check_core_req_variants( kbase_device *kbdev, kbasep_js_policy_cfs *policy_info )
+{
+ kbasep_js_device_data *js_devdata;
+ u32 i;
+ int j;
+
+ js_devdata = &kbdev->js_data;
+
+ for ( j = 0 ; j < kbdev->gpu_props.num_job_slots ; ++j )
+ {
+ base_jd_core_req job_core_req;
+ mali_bool found = MALI_FALSE;
+
+ job_core_req = js_devdata->js_reqs[j];
+ for ( i = 0; i < policy_info->num_core_req_variants ; ++i )
+ {
+ base_jd_core_req var_core_req;
+ var_core_req = policy_info->core_req_variants[i].core_req;
+
+ if ( (var_core_req & job_core_req) == job_core_req )
+ {
+ found = MALI_TRUE;
+ break;
+ }
+ }
+
+ /* Early-out on any failure */
+ OSK_ASSERT_MSG( found != MALI_FALSE,
+ "Job slot %d features 0x%x not matched by core_req_variants. "
+ "Rework core_req_variants and vairants_supported_<...>_state[] to match\n",
+ j,
+ job_core_req );
+ }
+}
+#endif
+
+STATIC void build_core_req_variants( kbase_device *kbdev, kbasep_js_policy_cfs *policy_info )
+{
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( policy_info != NULL );
+ CSTD_UNUSED( kbdev );
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8987))
+ {
+ OSK_ASSERT( NUM_CORE_REQ_VARIANTS_8987 <= KBASEP_JS_MAX_NR_CORE_REQ_VARIANTS );
+
+ /* Assume a static set of variants */
+ OSK_MEMCPY( policy_info->core_req_variants, core_req_variants_8987, sizeof(core_req_variants_8987) );
+
+ policy_info->num_core_req_variants = NUM_CORE_REQ_VARIANTS_8987;
+ }
+ else
+ {
+ OSK_ASSERT( NUM_CORE_REQ_VARIANTS <= KBASEP_JS_MAX_NR_CORE_REQ_VARIANTS );
+
+ /* Assume a static set of variants */
+ OSK_MEMCPY( policy_info->core_req_variants, core_req_variants, sizeof(core_req_variants) );
+
+ policy_info->num_core_req_variants = NUM_CORE_REQ_VARIANTS;
+ }
+
+ OSK_DEBUG_CODE( debug_check_core_req_variants( kbdev, policy_info ) );
+}
+
+
+STATIC void build_slot_lookups( kbase_device *kbdev, kbasep_js_policy_cfs *policy_info )
+{
+ u8 i;
+ const u32 *variants_supported_ss_for_this_hw = variants_supported_ss_state;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( policy_info != NULL );
+
+ OSK_ASSERT( kbdev->gpu_props.num_job_slots <= NELEMS(variants_supported_ss_state) );
+ OSK_ASSERT( kbdev->gpu_props.num_job_slots <= NELEMS(variants_supported_ss_allcore_state) );
+ OSK_ASSERT( kbdev->gpu_props.num_job_slots <= NELEMS(variants_supported_ss_state_8987) );
+ OSK_ASSERT( kbdev->gpu_props.num_job_slots <= NELEMS(variants_supported_nss_state) );
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8987))
+ {
+ variants_supported_ss_for_this_hw = variants_supported_ss_state_8987;
+ }
+
+ /* Given the static set of variants, provide a static set of lookups */
+ for ( i = 0; i < kbdev->gpu_props.num_job_slots; ++i )
+ {
+ set_slot_to_variant_lookup( policy_info->slot_to_variant_lookup_ss_state,
+ i,
+ variants_supported_ss_for_this_hw[i] );
+
+ set_slot_to_variant_lookup( policy_info->slot_to_variant_lookup_ss_allcore_state,
+ i,
+ variants_supported_ss_allcore_state[i] );
+
+ set_slot_to_variant_lookup( policy_info->slot_to_variant_lookup_nss_state,
+ i,
+ variants_supported_nss_state[i] );
+ }
+
+}
+
+STATIC mali_error cached_variant_idx_init( const kbasep_js_policy_cfs *policy_info, const kbase_context *kctx, kbase_jd_atom *atom )
+{
+ kbasep_js_policy_cfs_job *job_info;
+ u32 i;
+ base_jd_core_req job_core_req;
+ u32 job_device_nr;
+ kbase_context_flags ctx_flags;
+ const kbasep_js_kctx_info *js_kctx_info;
+ const kbase_device *kbdev;
+
+ OSK_ASSERT( policy_info != NULL );
+ OSK_ASSERT( kctx != NULL );
+ OSK_ASSERT( atom != NULL );
+
+ kbdev = CONTAINER_OF(policy_info, const kbase_device, js_data.policy.cfs);
+ job_info = &atom->sched_info.cfs;
+ job_core_req = atom->core_req;
+ job_device_nr = atom->device_nr;
+ js_kctx_info = &kctx->jctx.sched_info;
+ ctx_flags = js_kctx_info->ctx.flags;
+
+ /* Initial check for atoms targetting a specific coregroup */
+ if ( (job_core_req & BASE_JD_REQ_SPECIFIC_COHERENT_GROUP) != MALI_FALSE
+ && job_device_nr >= kbdev->gpu_props.num_core_groups )
+ {
+ /* device_nr exceeds the number of coregroups - not allowed by
+ * @ref base_jd_atom API contract */
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ /* Pick a core_req variant that matches us. Since they're ordered by least
+ * restrictive first, it picks the least restrictive variant */
+ for ( i = 0; i < policy_info->num_core_req_variants ; ++i )
+ {
+ base_jd_core_req var_core_req;
+ kbase_context_flags var_ctx_req;
+ u32 var_device_nr;
+ var_core_req = policy_info->core_req_variants[i].core_req;
+ var_ctx_req = policy_info->core_req_variants[i].ctx_req;
+ var_device_nr = policy_info->core_req_variants[i].device_nr;
+
+ if ( (var_core_req & job_core_req) == job_core_req
+ && (var_ctx_req & ctx_flags) == ctx_flags
+ && ((var_core_req & BASE_JD_REQ_SPECIFIC_COHERENT_GROUP)==MALI_FALSE || var_device_nr == job_device_nr ) )
+ {
+ job_info->cached_variant_idx = i;
+ return MALI_ERROR_NONE;
+ }
+ }
+
+ /* Could not find a matching requirement, this should only be caused by an
+ * attempt to attack the driver. */
+ return MALI_ERROR_FUNCTION_FAILED;
+}
+
+STATIC mali_bool dequeue_job( kbase_device *kbdev,
+ kbase_context *kctx,
+ u32 variants_supported,
+ kbase_jd_atom **katom_ptr,
+ int job_slot_idx)
+{
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_policy_cfs *policy_info;
+ kbasep_js_policy_cfs_ctx *ctx_info;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( katom_ptr != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ js_devdata = &kbdev->js_data;
+ policy_info = &js_devdata->policy.cfs;
+ ctx_info = &kctx->jctx.sched_info.runpool.policy_ctx.cfs;
+
+ /* Only submit jobs from contexts that are allowed */
+ if ( kbasep_js_is_submit_allowed( js_devdata, kctx ) != MALI_FALSE )
+ {
+ /* Check each variant in turn */
+ while ( variants_supported != 0 )
+ {
+ long variant_idx;
+ osk_dlist *job_list;
+ variant_idx = osk_find_first_set_bit( variants_supported );
+ job_list = &ctx_info->job_list_head[variant_idx];
+
+ if ( OSK_DLIST_IS_EMPTY( job_list ) == MALI_FALSE )
+ {
+ /* Found a context with a matching job */
+ {
+ kbase_jd_atom *front_atom = OSK_DLIST_FRONT( job_list, kbase_jd_atom, sched_info.cfs.list );
+ KBASE_TRACE_ADD_SLOT( kbdev, JS_POLICY_DEQUEUE_JOB, front_atom->kctx, front_atom->user_atom,
+ front_atom->jc, job_slot_idx );
+ }
+ *katom_ptr = OSK_DLIST_POP_FRONT( job_list, kbase_jd_atom, sched_info.cfs.list );
+
+ (*katom_ptr)->sched_info.cfs.ticks = 0;
+
+ /* Put this context at the back of the Run Pool */
+ OSK_DLIST_REMOVE( &policy_info->scheduled_ctxs_head,
+ kctx,
+ jctx.sched_info.runpool.policy_ctx.cfs.list );
+ OSK_DLIST_PUSH_BACK( &policy_info->scheduled_ctxs_head,
+ kctx,
+ kbase_context,
+ jctx.sched_info.runpool.policy_ctx.cfs.list );
+
+ return MALI_TRUE;
+ }
+
+ variants_supported &= ~(1u << variant_idx);
+ }
+ /* All variants checked by here */
+ }
+
+ /* The context does not have a matching job */
+
+ return MALI_FALSE;
+}
+
+/**
+ * Hold the runpool_irq spinlock for this
+ */
+OSK_STATIC_INLINE mali_bool timer_callback_should_run( kbase_device *kbdev )
+{
+ kbasep_js_device_data *js_devdata;
+ s8 nr_running_ctxs;
+
+ OSK_ASSERT(kbdev != NULL);
+ js_devdata = &kbdev->js_data;
+
+ /* nr_user_contexts_running is updated with the runpool_mutex. However, the
+ * locking in the caller gives us a barrier that ensures nr_user_contexts is
+ * up-to-date for reading */
+ nr_running_ctxs = js_devdata->nr_user_contexts_running;
+
+#if MALI_DEBUG
+ if(js_devdata->softstop_always)
+ {
+ /* Debug support for allowing soft-stop on a single context */
+ return MALI_TRUE;
+ }
+#endif /* MALI_DEBUG */
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_9435))
+ {
+ /* Timeouts would have to be 4x longer (due to micro-architectural design)
+ * to support OpenCL conformance tests, so only run the timer when there's:
+ * - 2 or more CL contexts
+ * - 1 or more GLES contexts
+ *
+ * NOTE: We will treat a context that has both Compute and Non-Compute jobs
+ * will be treated as an OpenCL context (hence, we don't check
+ * KBASEP_JS_CTX_ATTR_NON_COMPUTE).
+ */
+ {
+ s8 nr_compute_ctxs = kbasep_js_ctx_attr_count_on_runpool( kbdev, KBASEP_JS_CTX_ATTR_COMPUTE );
+ s8 nr_noncompute_ctxs = nr_running_ctxs - nr_compute_ctxs;
+
+ return (mali_bool)( nr_compute_ctxs >= 2 || nr_noncompute_ctxs > 0 );
+ }
+ }
+ else
+ {
+ /* Run the timer callback whenever you have at least 1 context */
+ return (mali_bool)(nr_running_ctxs > 0);
+ }
+}
+
+static void timer_callback(void *data)
+{
+ kbase_device *kbdev = (kbase_device*)data;
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_policy_cfs *policy_info;
+ int s;
+ osk_error osk_err;
+ mali_bool reset_needed = MALI_FALSE;
+
+ OSK_ASSERT(kbdev != NULL);
+
+ js_devdata = &kbdev->js_data;
+ policy_info = &js_devdata->policy.cfs;
+
+ /* Loop through the slots */
+ for(s=0; s<kbdev->gpu_props.num_job_slots; s++)
+ {
+ kbase_jm_slot *slot = kbase_job_slot_lock(kbdev, s);
+ kbase_jd_atom *atom = NULL;
+
+ if (kbasep_jm_nr_jobs_submitted(slot) > 0)
+ {
+ atom = kbasep_jm_peek_idx_submit_slot(slot, 0);
+ OSK_ASSERT( atom != NULL );
+
+ if ( kbasep_jm_is_dummy_workaround_job( kbdev, atom ) != MALI_FALSE )
+ {
+ /* Prevent further use of the atom - never cause a soft-stop, hard-stop, or a GPU reset due to it. */
+ atom = NULL;
+ }
+ }
+
+ if ( atom != NULL )
+ {
+ /* The current version of the model doesn't support Soft-Stop */
+ if (!kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_5736))
+ {
+ u32 ticks = atom->sched_info.cfs.ticks ++;
+
+#if !CINSTR_DUMPING_ENABLED
+ if ( (atom->core_req & BASE_JD_REQ_NSS) == 0 )
+ {
+ /* Job is Soft-Stoppable */
+ if (ticks == js_devdata->soft_stop_ticks)
+ {
+ /* Job has been scheduled for at least js_devdata->soft_stop_ticks ticks.
+ * Soft stop the slot so we can run other jobs.
+ */
+ OSK_PRINT_INFO( OSK_BASE_JM, "Soft-stop" );
+
+#if KBASE_DISABLE_SCHEDULING_SOFT_STOPS == 0
+ kbase_job_slot_softstop(kbdev, s, atom);
+#endif
+ }
+ else if (ticks == js_devdata->hard_stop_ticks_ss)
+ {
+ /* Job has been scheduled for at least js_devdata->hard_stop_ticks_ss ticks.
+ * It should have been soft-stopped by now. Hard stop the slot.
+ */
+#if KBASE_DISABLE_SCHEDULING_HARD_STOPS == 0
+ OSK_PRINT_WARN(OSK_BASE_JM, "JS: Job Hard-Stopped (took more than %lu ticks at %lu ms/tick)", ticks, js_devdata->scheduling_tick_ns/1000000u );
+ kbase_job_slot_hardstop(atom->kctx, s, atom);
+#endif
+ }
+ else if (ticks == js_devdata->gpu_reset_ticks_ss)
+ {
+ /* Job has been scheduled for at least js_devdata->gpu_reset_ticks_ss ticks.
+ * It should have left the GPU by now. Signal that the GPU needs to be reset.
+ */
+ reset_needed = MALI_TRUE;
+ }
+ }
+ else
+#endif /* !CINSTR_DUMPING_ENABLED */
+ /* NOTE: During CINSTR_DUMPING_ENABLED, we use the NSS-timeouts for *all* atoms,
+ * which makes the hard-stop and GPU reset timeout much longer. We also ensure
+ * that we don't soft-stop at all.
+ *
+ * Otherwise, this next block is only used for NSS-atoms */
+ {
+ /* Job is Non Soft-Stoppable */
+ if (ticks == js_devdata->soft_stop_ticks)
+ {
+ /* Job has been scheduled for at least js_devdata->soft_stop_ticks.
+ * Let's try to soft-stop it even if it's supposed to be NSS.
+ */
+ OSK_PRINT_INFO( OSK_BASE_JM, "Soft-stop" );
+
+#if (KBASE_DISABLE_SCHEDULING_SOFT_STOPS == 0) && (CINSTR_DUMPING_ENABLED == 0)
+ kbase_job_slot_softstop(kbdev, s, atom);
+#endif
+ }
+ else if (ticks == js_devdata->hard_stop_ticks_nss)
+ {
+ /* Job has been scheduled for at least js_devdata->hard_stop_ticks_nss ticks.
+ * Hard stop the slot.
+ */
+#if KBASE_DISABLE_SCHEDULING_HARD_STOPS == 0
+ OSK_PRINT_WARN(OSK_BASE_JM, "JS: Job Hard-Stopped (took more than %lu ticks at %lu ms/tick)", ticks, js_devdata->scheduling_tick_ns/1000000u );
+ kbase_job_slot_hardstop(atom->kctx, s, atom);
+#endif
+ }
+ else if (ticks == js_devdata->gpu_reset_ticks_nss)
+ {
+ /* Job has been scheduled for at least js_devdata->gpu_reset_ticks_nss ticks.
+ * It should have left the GPU by now. Signal that the GPU needs to be reset.
+ */
+ reset_needed = MALI_TRUE;
+ }
+ }
+ }
+ }
+ kbase_job_slot_unlock(kbdev, s);
+ }
+
+ if (reset_needed)
+ {
+ OSK_PRINT_WARN(OSK_BASE_JM, "JS: Job has been on the GPU for too long");
+ if (kbase_prepare_to_reset_gpu(kbdev))
+ {
+ kbase_reset_gpu(kbdev);
+ }
+ }
+
+ /* the timer is re-issued if there is contexts in the run-pool */
+ osk_spinlock_irq_lock(&js_devdata->runpool_irq.lock);
+
+ if (timer_callback_should_run(kbdev) != MALI_FALSE)
+ {
+ osk_err = osk_timer_start_ns(&policy_info->timer, js_devdata->scheduling_tick_ns);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ policy_info->timer_running = MALI_FALSE;
+ }
+ }
+ else
+ {
+ KBASE_TRACE_ADD( kbdev, JS_POLICY_TIMER_END, NULL, NULL, 0u, 0u );
+ policy_info->timer_running = MALI_FALSE;
+ }
+
+ osk_spinlock_irq_unlock(&js_devdata->runpool_irq.lock);
+}
+
+/*
+ * Non-private functions
+ */
+
+mali_error kbasep_js_policy_init( kbase_device *kbdev )
+{
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_policy_cfs *policy_info;
+
+ OSK_ASSERT( kbdev != NULL );
+ js_devdata = &kbdev->js_data;
+ policy_info = &js_devdata->policy.cfs;
+
+ OSK_DLIST_INIT( &policy_info->ctx_queue_head );
+ OSK_DLIST_INIT( &policy_info->scheduled_ctxs_head );
+ OSK_DLIST_INIT( &policy_info->ctx_rt_queue_head );
+
+ if (osk_timer_init(&policy_info->timer) != OSK_ERR_NONE)
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ osk_timer_callback_set( &policy_info->timer, timer_callback, kbdev );
+
+ policy_info->timer_running = MALI_FALSE;
+
+ policy_info->head_runtime_us = 0;
+
+ /* Build up the core_req variants */
+ build_core_req_variants( kbdev, policy_info );
+ /* Build the slot to variant lookups */
+ build_slot_lookups(kbdev, policy_info );
+
+ return MALI_ERROR_NONE;
+}
+
+void kbasep_js_policy_term( kbasep_js_policy *js_policy )
+{
+ kbasep_js_policy_cfs *policy_info;
+
+ OSK_ASSERT( js_policy != NULL );
+ policy_info = &js_policy->cfs;
+
+ /* ASSERT that there are no contexts queued */
+ OSK_ASSERT( OSK_DLIST_IS_EMPTY( &policy_info->ctx_queue_head ) != MALI_FALSE );
+ /* ASSERT that there are no contexts scheduled */
+ OSK_ASSERT( OSK_DLIST_IS_EMPTY( &policy_info->scheduled_ctxs_head ) != MALI_FALSE );
+
+ /* ASSERT that there are no contexts queued */
+ OSK_ASSERT( OSK_DLIST_IS_EMPTY( &policy_info->ctx_rt_queue_head ) != MALI_FALSE );
+
+ osk_timer_stop(&policy_info->timer);
+ osk_timer_term(&policy_info->timer);
+}
+
+mali_error kbasep_js_policy_init_ctx( kbase_device *kbdev, kbase_context *kctx )
+{
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_policy_cfs_ctx *ctx_info;
+ kbasep_js_policy_cfs *policy_info;
+ osk_process_priority prio;
+ u32 i;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ js_devdata = &kbdev->js_data;
+ policy_info = &kbdev->js_data.policy.cfs;
+ ctx_info = &kctx->jctx.sched_info.runpool.policy_ctx.cfs;
+
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, JS_POLICY_INIT_CTX, kctx, NULL, 0u,
+ kbasep_js_policy_trace_get_refcnt( kbdev, kctx ));
+
+ for ( i = 0 ; i < policy_info->num_core_req_variants ; ++i )
+ {
+ OSK_DLIST_INIT( &ctx_info->job_list_head[i] );
+ }
+
+ osk_get_process_priority(&prio);
+ ctx_info->process_rt_policy = prio.is_realtime;
+ ctx_info->process_priority = prio.priority;
+ ctx_info->bag_total_priority = 0;
+ ctx_info->bag_total_nr_atoms = 0;
+
+ /* Initial runtime (relative to least-run context runtime)
+ *
+ * This uses the Policy Queue's most up-to-date head_runtime_us by using the
+ * queue mutex to issue memory barriers - also ensure future updates to
+ * head_runtime_us occur strictly after this context is initialized */
+ osk_mutex_lock( &js_devdata->queue_mutex );
+
+ /* No need to hold the the runpool_irq.lock here, because we're initializing
+ * the value, and the context is definitely not being updated in the
+ * runpool at this point. The queue_mutex ensures the memory barrier. */
+ ctx_info->runtime_us = policy_info->head_runtime_us +
+ priority_weight(ctx_info,
+ (u64)js_devdata->cfs_ctx_runtime_init_slices * (u64)(js_devdata->ctx_timeslice_ns/1000u));
+
+ osk_mutex_unlock( &js_devdata->queue_mutex );
+
+ return MALI_ERROR_NONE;
+}
+
+void kbasep_js_policy_term_ctx( kbasep_js_policy *js_policy, kbase_context *kctx )
+{
+ kbasep_js_policy_cfs_ctx *ctx_info;
+ kbasep_js_policy_cfs *policy_info;
+ u32 i;
+
+ OSK_ASSERT( js_policy != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ policy_info = &js_policy->cfs;
+ ctx_info = &kctx->jctx.sched_info.runpool.policy_ctx.cfs;
+
+ {
+ kbase_device *kbdev = CONTAINER_OF( js_policy, kbase_device, js_data.policy );
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, JS_POLICY_TERM_CTX, kctx, NULL, 0u,
+ kbasep_js_policy_trace_get_refcnt( kbdev, kctx ));
+ }
+
+ /* ASSERT that no jobs are present */
+ for ( i = 0 ; i < policy_info->num_core_req_variants ; ++i )
+ {
+ OSK_ASSERT( OSK_DLIST_IS_EMPTY( &ctx_info->job_list_head[i] ) != MALI_FALSE );
+ }
+
+ /* No work to do */
+}
+
+
+/*
+ * Context Management
+ */
+
+void kbasep_js_policy_enqueue_ctx( kbasep_js_policy *js_policy, kbase_context *kctx )
+{
+ kbasep_js_policy_cfs *policy_info;
+ kbasep_js_policy_cfs_ctx *ctx_info;
+ kbase_context *list_kctx = NULL;
+ kbasep_js_device_data *js_devdata;
+ osk_dlist *queue_head;
+
+ OSK_ASSERT( js_policy != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ policy_info = &js_policy->cfs;
+ ctx_info = &kctx->jctx.sched_info.runpool.policy_ctx.cfs;
+ js_devdata = CONTAINER_OF( js_policy, kbasep_js_device_data, policy );
+
+ {
+ kbase_device *kbdev = CONTAINER_OF( js_policy, kbase_device, js_data.policy );
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, JS_POLICY_ENQUEUE_CTX, kctx, NULL, 0u,
+ kbasep_js_policy_trace_get_refcnt( kbdev, kctx ));
+ }
+
+ /* ASSERT about scheduled-ness/queued-ness */
+ kbasep_js_debug_check( policy_info, kctx, KBASEP_JS_CHECK_NOTQUEUED );
+
+ /* Clamp the runtime to prevent DoS attacks through "stored-up" runtime */
+ if (policy_info->head_runtime_us > ctx_info->runtime_us
+ + (u64)js_devdata->cfs_ctx_runtime_min_slices * (u64)(js_devdata->ctx_timeslice_ns/1000u))
+ {
+ /* No need to hold the the runpool_irq.lock here, because we're essentially
+ * initializing the value, and the context is definitely not being updated in the
+ * runpool at this point. The queue_mutex held by the caller ensures the memory
+ * barrier. */
+ ctx_info->runtime_us = policy_info->head_runtime_us
+ - (u64)js_devdata->cfs_ctx_runtime_min_slices * (u64)(js_devdata->ctx_timeslice_ns/1000u);
+ }
+
+ /* Find the position where the context should be enqueued */
+ if(ctx_info->process_rt_policy)
+ {
+ queue_head = &policy_info->ctx_rt_queue_head;
+ }
+ else
+ {
+ queue_head = &policy_info->ctx_queue_head;
+ }
+
+ OSK_DLIST_FOREACH( queue_head,
+ kbase_context,
+ jctx.sched_info.runpool.policy_ctx.cfs.list,
+ list_kctx )
+ {
+ kbasep_js_policy_cfs_ctx *list_ctx_info;
+ list_ctx_info = &list_kctx->jctx.sched_info.runpool.policy_ctx.cfs;
+
+ if ( (kctx->jctx.sched_info.ctx.flags & KBASE_CTX_FLAG_PRIVILEGED) != 0 )
+ {
+ break;
+ }
+
+ if ( (list_ctx_info->runtime_us > ctx_info->runtime_us) &&
+ ((list_kctx->jctx.sched_info.ctx.flags & KBASE_CTX_FLAG_PRIVILEGED) == 0) )
+ {
+ break;
+ }
+ }
+
+ /* Add the context to the queue */
+ if (OSK_DLIST_IS_VALID( list_kctx, jctx.sched_info.runpool.policy_ctx.cfs.list ) == MALI_TRUE)
+ {
+ OSK_DLIST_INSERT_BEFORE( queue_head,
+ kctx,
+ list_kctx,
+ kbase_context,
+ jctx.sched_info.runpool.policy_ctx.cfs.list );
+ }
+ else
+ {
+ OSK_DLIST_PUSH_BACK( queue_head,
+ kctx,
+ kbase_context,
+ jctx.sched_info.runpool.policy_ctx.cfs.list );
+ }
+}
+
+mali_bool kbasep_js_policy_dequeue_head_ctx( kbasep_js_policy *js_policy, kbase_context **kctx_ptr )
+{
+ kbasep_js_policy_cfs *policy_info;
+ kbase_context *head_ctx;
+ osk_dlist *queue_head;
+
+ OSK_ASSERT( js_policy != NULL );
+ OSK_ASSERT( kctx_ptr != NULL );
+
+ policy_info = &js_policy->cfs;
+
+ /* attempt to dequeue from the 'realttime' queue first */
+ if ( OSK_DLIST_IS_EMPTY( &policy_info->ctx_rt_queue_head ) != MALI_FALSE )
+ {
+ if ( OSK_DLIST_IS_EMPTY( &policy_info->ctx_queue_head ) != MALI_FALSE )
+ {
+ /* Nothing to dequeue */
+ return MALI_FALSE;
+ }
+ else
+ {
+ queue_head = &policy_info->ctx_queue_head;
+ }
+ }
+ else
+ {
+ queue_head = &policy_info->ctx_rt_queue_head;
+ }
+
+ /* Contexts are dequeued from the front of the queue */
+ *kctx_ptr = OSK_DLIST_POP_FRONT( queue_head,
+ kbase_context,
+ jctx.sched_info.runpool.policy_ctx.cfs.list );
+
+ {
+ kbase_device *kbdev = CONTAINER_OF( js_policy, kbase_device, js_data.policy );
+ kbase_context *kctx = *kctx_ptr;
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, JS_POLICY_DEQUEUE_HEAD_CTX, kctx, NULL, 0u,
+ kbasep_js_policy_trace_get_refcnt( kbdev, kctx ));
+ }
+
+
+ /* Update the head runtime */
+ head_ctx = OSK_DLIST_FRONT( queue_head,
+ kbase_context,
+ jctx.sched_info.runpool.policy_ctx.cfs.list );
+ if (OSK_DLIST_IS_VALID( head_ctx, jctx.sched_info.runpool.policy_ctx.cfs.list ) == MALI_TRUE)
+ {
+ /* No need to hold the the runpool_irq.lock here for reading - the
+ * context is definitely not being updated in the runpool at this
+ * point. The queue_mutex held by the caller ensures the memory barrier. */
+ u64 head_runtime = head_ctx->jctx.sched_info.runpool.policy_ctx.cfs.runtime_us;
+
+ if (head_runtime > policy_info->head_runtime_us)
+ {
+ policy_info->head_runtime_us = head_runtime;
+ }
+ }
+
+ return MALI_TRUE;
+}
+
+mali_bool kbasep_js_policy_try_evict_ctx( kbasep_js_policy *js_policy, kbase_context *kctx )
+{
+ kbasep_js_policy_cfs_ctx *ctx_info;
+ kbasep_js_policy_cfs *policy_info;
+ mali_bool is_present;
+ osk_dlist *queue_head;
+ osk_dlist *qhead;
+
+ OSK_ASSERT( js_policy != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ policy_info = &js_policy->cfs;
+ ctx_info = &kctx->jctx.sched_info.runpool.policy_ctx.cfs;
+
+ if(ctx_info->process_rt_policy)
+ {
+ queue_head = &policy_info->ctx_rt_queue_head;
+ }
+ else
+ {
+ queue_head = &policy_info->ctx_queue_head;
+ }
+ qhead = queue_head;
+
+ is_present = OSK_DLIST_MEMBER_OF( qhead,
+ kctx,
+ jctx.sched_info.runpool.policy_ctx.cfs.list );
+
+ {
+ kbase_device *kbdev = CONTAINER_OF( js_policy, kbase_device, js_data.policy );
+ KBASE_TRACE_ADD_REFCOUNT_INFO( kbdev, JS_POLICY_TRY_EVICT_CTX, kctx, NULL, 0u,
+ kbasep_js_policy_trace_get_refcnt( kbdev, kctx ), is_present);
+ }
+
+ if ( is_present != MALI_FALSE )
+ {
+ kbase_context *head_ctx;
+ qhead = queue_head;
+ /* Remove the context */
+ OSK_DLIST_REMOVE( qhead,
+ kctx,
+ jctx.sched_info.runpool.policy_ctx.cfs.list );
+
+ qhead = queue_head;
+ /* Update the head runtime */
+ head_ctx = OSK_DLIST_FRONT( qhead,
+ kbase_context,
+ jctx.sched_info.runpool.policy_ctx.cfs.list );
+ if (OSK_DLIST_IS_VALID( head_ctx, jctx.sched_info.runpool.policy_ctx.cfs.list ) == MALI_TRUE)
+ {
+ /* No need to hold the the runpool_irq.lock here for reading - the
+ * context is definitely not being updated in the runpool at this
+ * point. The queue_mutex held by the caller ensures the memory barrier. */
+ u64 head_runtime = head_ctx->jctx.sched_info.runpool.policy_ctx.cfs.runtime_us;
+
+ if (head_runtime > policy_info->head_runtime_us)
+ {
+ policy_info->head_runtime_us = head_runtime;
+ }
+ }
+ }
+
+ return is_present;
+}
+
+void kbasep_js_policy_kill_all_ctx_jobs( kbasep_js_policy *js_policy, kbase_context *kctx )
+{
+ kbasep_js_policy_cfs *policy_info;
+ kbasep_js_policy_cfs_ctx *ctx_info;
+ u32 i;
+
+ OSK_ASSERT( js_policy != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ policy_info = &js_policy->cfs;
+ ctx_info = &kctx->jctx.sched_info.runpool.policy_ctx.cfs;
+
+ {
+ kbase_device *kbdev = CONTAINER_OF( js_policy, kbase_device, js_data.policy );
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, JS_POLICY_KILL_ALL_CTX_JOBS, kctx, NULL, 0u,
+ kbasep_js_policy_trace_get_refcnt( kbdev, kctx ));
+ }
+
+ /* Kill jobs on each variant in turn */
+ for ( i = 0; i < policy_info->num_core_req_variants; ++i )
+ {
+ osk_dlist *job_list;
+ job_list = &ctx_info->job_list_head[i];
+
+ /* Call kbase_jd_cancel() on all kbase_jd_atoms in this list, whilst removing them from the list */
+ OSK_DLIST_EMPTY_LIST( job_list, kbase_jd_atom, sched_info.cfs.list, kbase_jd_cancel );
+ }
+
+}
+
+void kbasep_js_policy_runpool_add_ctx( kbasep_js_policy *js_policy, kbase_context *kctx )
+{
+ kbasep_js_policy_cfs *policy_info;
+ kbasep_js_device_data *js_devdata;
+ kbase_device *kbdev;
+ osk_error osk_err;
+
+ OSK_ASSERT( js_policy != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ policy_info = &js_policy->cfs;
+ js_devdata = CONTAINER_OF( js_policy, kbasep_js_device_data, policy );
+ kbdev = CONTAINER_OF( js_policy, kbase_device, js_data.policy );
+
+ {
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, JS_POLICY_RUNPOOL_ADD_CTX, kctx, NULL, 0u,
+ kbasep_js_policy_trace_get_refcnt_nolock( kbdev, kctx ));
+ }
+
+ /* ASSERT about scheduled-ness/queued-ness */
+ kbasep_js_debug_check( policy_info, kctx, KBASEP_JS_CHECK_NOTSCHEDULED );
+
+ /* All enqueued contexts go to the back of the runpool */
+ OSK_DLIST_PUSH_BACK( &policy_info->scheduled_ctxs_head,
+ kctx,
+ kbase_context,
+ jctx.sched_info.runpool.policy_ctx.cfs.list );
+
+ if ( timer_callback_should_run(kbdev) != MALI_FALSE
+ && policy_info->timer_running == MALI_FALSE )
+ {
+ osk_err = osk_timer_start_ns(&policy_info->timer, js_devdata->scheduling_tick_ns);
+ if (OSK_ERR_NONE == osk_err)
+ {
+ kbase_device *kbdev = CONTAINER_OF( js_policy, kbase_device, js_data.policy );
+ KBASE_TRACE_ADD( kbdev, JS_POLICY_TIMER_START, NULL, NULL, 0u, 0u );
+ policy_info->timer_running = MALI_TRUE;
+ }
+ }
+}
+
+void kbasep_js_policy_runpool_remove_ctx( kbasep_js_policy *js_policy, kbase_context *kctx )
+{
+ kbasep_js_policy_cfs *policy_info;
+
+ OSK_ASSERT( js_policy != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ policy_info = &js_policy->cfs;
+
+ {
+ kbase_device *kbdev = CONTAINER_OF( js_policy, kbase_device, js_data.policy );
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, JS_POLICY_RUNPOOL_REMOVE_CTX, kctx, NULL, 0u,
+ kbasep_js_policy_trace_get_refcnt_nolock( kbdev, kctx ));
+ }
+
+ /* ASSERT about scheduled-ness/queued-ness */
+ kbasep_js_debug_check( policy_info, kctx, KBASEP_JS_CHECK_SCHEDULED );
+
+ /* No searching or significant list maintenance required to remove this context */
+ OSK_DLIST_REMOVE( &policy_info->scheduled_ctxs_head,
+ kctx,
+ jctx.sched_info.runpool.policy_ctx.cfs.list );
+}
+
+mali_bool kbasep_js_policy_should_remove_ctx( kbasep_js_policy *js_policy, kbase_context *kctx )
+{
+ kbasep_js_policy_cfs_ctx *ctx_info;
+ kbasep_js_policy_cfs *policy_info;
+ kbase_context *head_ctx;
+ kbasep_js_device_data *js_devdata;
+ osk_dlist *queue_head;
+
+ OSK_ASSERT( js_policy != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ policy_info = &js_policy->cfs;
+ ctx_info = &kctx->jctx.sched_info.runpool.policy_ctx.cfs;
+ js_devdata = CONTAINER_OF( js_policy, kbasep_js_device_data, policy );
+
+ if(ctx_info->process_rt_policy)
+ {
+ queue_head = &policy_info->ctx_rt_queue_head;
+ }
+ else
+ {
+ queue_head = &policy_info->ctx_queue_head;
+ }
+
+ head_ctx = OSK_DLIST_FRONT( queue_head,
+ kbase_context,
+ jctx.sched_info.runpool.policy_ctx.cfs.list );
+ if (OSK_DLIST_IS_VALID( head_ctx, jctx.sched_info.runpool.policy_ctx.cfs.list ) == MALI_TRUE)
+ {
+ u64 head_runtime_us = head_ctx->jctx.sched_info.runpool.policy_ctx.cfs.runtime_us;
+
+ if ((head_runtime_us + priority_weight(ctx_info, (u64)(js_devdata->ctx_timeslice_ns/1000u)))
+ < ctx_info->runtime_us)
+ {
+ /* The context is scheduled out if it's not the least-run context anymore.
+ * The "real" head runtime is used instead of the cached runtime so the current
+ * context is not scheduled out when there is less contexts than address spaces.
+ */
+ return MALI_TRUE;
+ }
+ }
+
+ return MALI_FALSE;
+}
+
+/*
+ * Job Chain Management
+ */
+
+mali_error kbasep_js_policy_init_job( const kbasep_js_policy *js_policy, const kbase_context *kctx, kbase_jd_atom *katom )
+{
+ const kbasep_js_policy_cfs *policy_info;
+
+ OSK_ASSERT( js_policy != NULL );
+ OSK_ASSERT( katom != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ policy_info = &js_policy->cfs;
+
+ /* Determine the job's index into the job list head, will return error if the
+ * atom is malformed and so is reported. */
+ return cached_variant_idx_init( policy_info, kctx, katom );
+}
+
+void kbasep_js_policy_term_job( const kbasep_js_policy *js_policy, const kbase_context *kctx, kbase_jd_atom *katom )
+{
+ kbasep_js_policy_cfs_job *job_info;
+ const kbasep_js_policy_cfs_ctx *ctx_info;
+
+ OSK_ASSERT( js_policy != NULL );
+ CSTD_UNUSED(js_policy);
+ OSK_ASSERT( katom != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ job_info = &katom->sched_info.cfs;
+ ctx_info = &kctx->jctx.sched_info.runpool.policy_ctx.cfs;
+
+ /* We need not do anything, so we just ASSERT that this job was correctly removed from the relevant lists */
+ OSK_ASSERT( OSK_DLIST_MEMBER_OF( &ctx_info->job_list_head[job_info->cached_variant_idx],
+ katom,
+ sched_info.cfs.list ) == MALI_FALSE );
+}
+
+void kbasep_js_policy_register_job( kbasep_js_policy *js_policy, kbase_context *kctx, kbase_jd_atom *katom )
+{
+ kbasep_js_policy_cfs_ctx *ctx_info;
+
+ OSK_ASSERT( js_policy != NULL );
+ OSK_ASSERT( katom != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ ctx_info = &kctx->jctx.sched_info.runpool.policy_ctx.cfs;
+
+ /* Adjust context priority to include the new job */
+ ctx_info->bag_total_nr_atoms++;
+ ctx_info->bag_total_priority += katom->nice_prio;
+
+ /* Get average priority and convert to NICE range -20..19 */
+ if(ctx_info->bag_total_nr_atoms)
+ {
+ ctx_info->bag_priority = (ctx_info->bag_total_priority / ctx_info->bag_total_nr_atoms) - 20;
+ }
+}
+
+void kbasep_js_policy_deregister_job( kbasep_js_policy *js_policy, kbase_context *kctx, kbase_jd_atom *katom )
+{
+ kbasep_js_policy_cfs_ctx *ctx_info;
+
+ OSK_ASSERT( js_policy != NULL );
+ CSTD_UNUSED(js_policy);
+ OSK_ASSERT( katom != NULL );
+ OSK_ASSERT( kctx != NULL );
+
+ ctx_info = &kctx->jctx.sched_info.runpool.policy_ctx.cfs;
+
+ /* Adjust context priority to no longer include removed job */
+ OSK_ASSERT(ctx_info->bag_total_nr_atoms > 0);
+ ctx_info->bag_total_nr_atoms--;
+ ctx_info->bag_total_priority -= katom->nice_prio;
+ OSK_ASSERT(ctx_info->bag_total_priority >= 0);
+
+ /* Get average priority and convert to NICE range -20..19 */
+ if(ctx_info->bag_total_nr_atoms)
+ {
+ ctx_info->bag_priority = (ctx_info->bag_total_priority / ctx_info->bag_total_nr_atoms) - 20;
+ }
+}
+KBASE_EXPORT_TEST_API(kbasep_js_policy_deregister_job)
+
+mali_bool kbasep_js_policy_dequeue_job( kbase_device *kbdev,
+ int job_slot_idx,
+ kbase_jd_atom **katom_ptr )
+{
+ kbasep_js_device_data *js_devdata;
+ kbasep_js_policy_cfs *policy_info;
+ kbase_context *kctx;
+ u32 variants_supported;
+
+ OSK_ASSERT( kbdev != NULL );
+ OSK_ASSERT( katom_ptr != NULL );
+ OSK_ASSERT( job_slot_idx < BASE_JM_MAX_NR_SLOTS );
+
+ js_devdata = &kbdev->js_data;
+ policy_info = &js_devdata->policy.cfs;
+
+ /* Get the variants for this slot */
+ if ( kbasep_js_ctx_attr_is_attr_on_runpool( kbdev, KBASEP_JS_CTX_ATTR_NSS ) != MALI_FALSE )
+ {
+ /* NSS-state */
+ variants_supported = get_slot_to_variant_lookup( policy_info->slot_to_variant_lookup_nss_state, job_slot_idx );
+ }
+ else if ( kbdev->gpu_props.num_core_groups > 1
+ && kbasep_js_ctx_attr_is_attr_on_runpool( kbdev, KBASEP_JS_CTX_ATTR_COMPUTE_ALL_CORES ) != MALI_FALSE )
+ {
+ /* SS-allcore state, and there's more than one coregroup */
+ variants_supported = get_slot_to_variant_lookup( policy_info->slot_to_variant_lookup_ss_allcore_state, job_slot_idx );
+ }
+ else
+ {
+ /* SS-state */
+ variants_supported = get_slot_to_variant_lookup( policy_info->slot_to_variant_lookup_ss_state, job_slot_idx );
+ }
+
+ /* First pass through the runpool we consider the realtime priority jobs */
+ OSK_DLIST_FOREACH( &policy_info->scheduled_ctxs_head,
+ kbase_context,
+ jctx.sched_info.runpool.policy_ctx.cfs.list,
+ kctx )
+ {
+ if(kctx->jctx.sched_info.runpool.policy_ctx.cfs.process_rt_policy)
+ {
+ if(dequeue_job(kbdev, kctx, variants_supported, katom_ptr, job_slot_idx))
+ {
+ /* Realtime policy job matched */
+ return MALI_TRUE;
+ }
+ }
+ }
+
+ /* Second pass through the runpool we consider the non-realtime priority jobs */
+ OSK_DLIST_FOREACH( &policy_info->scheduled_ctxs_head,
+ kbase_context,
+ jctx.sched_info.runpool.policy_ctx.cfs.list,
+ kctx )
+ {
+ if(kctx->jctx.sched_info.runpool.policy_ctx.cfs.process_rt_policy == MALI_FALSE)
+ {
+ if(dequeue_job(kbdev, kctx, variants_supported, katom_ptr, job_slot_idx))
+ {
+ /* Non-realtime policy job matched */
+ return MALI_TRUE;
+ }
+ }
+ }
+
+ /* By this point, no contexts had a matching job */
+ return MALI_FALSE;
+}
+
+mali_bool kbasep_js_policy_dequeue_job_irq( kbase_device *kbdev,
+ int job_slot_idx,
+ kbase_jd_atom **katom_ptr )
+{
+ /* IRQ and non-IRQ variants of this are the same (though, the IRQ variant could be made faster) */
+
+ /* KBASE_TRACE_ADD_SLOT( kbdev, JS_POLICY_DEQUEUE_JOB_IRQ, NULL, NULL, 0u,
+ job_slot_idx); */
+ return kbasep_js_policy_dequeue_job( kbdev, job_slot_idx, katom_ptr );
+}
+
+
+void kbasep_js_policy_enqueue_job( kbasep_js_policy *js_policy, kbase_jd_atom *katom )
+{
+ kbasep_js_policy_cfs_job *job_info;
+ kbasep_js_policy_cfs_ctx *ctx_info;
+ kbase_context *parent_ctx;
+
+ OSK_ASSERT( js_policy != NULL );
+ OSK_ASSERT( katom != NULL );
+ parent_ctx = katom->kctx;
+ OSK_ASSERT( parent_ctx != NULL );
+
+ job_info = &katom->sched_info.cfs;
+ ctx_info = &parent_ctx->jctx.sched_info.runpool.policy_ctx.cfs;
+
+ {
+ kbase_device *kbdev = CONTAINER_OF( js_policy, kbase_device, js_data.policy );
+ KBASE_TRACE_ADD( kbdev, JS_POLICY_ENQUEUE_JOB, katom->kctx, katom->user_atom, katom->jc,
+ 0 );
+ }
+
+ OSK_DLIST_PUSH_BACK( &ctx_info->job_list_head[job_info->cached_variant_idx],
+ katom,
+ kbase_jd_atom,
+ sched_info.cfs.list );
+}
+
+void kbasep_js_policy_log_job_result( kbasep_js_policy *js_policy, kbase_jd_atom *katom, u32 time_spent_us )
+{
+ kbasep_js_policy_cfs_ctx *ctx_info;
+ kbase_context *parent_ctx;
+ OSK_ASSERT( js_policy != NULL );
+ OSK_ASSERT( katom != NULL );
+ CSTD_UNUSED( js_policy );
+
+ parent_ctx = katom->kctx;
+ OSK_ASSERT( parent_ctx != NULL );
+
+ ctx_info = &parent_ctx->jctx.sched_info.runpool.policy_ctx.cfs;
+
+ ctx_info->runtime_us += priority_weight(ctx_info, time_spent_us);
+}
+
+mali_bool kbasep_js_policy_ctx_has_priority( kbasep_js_policy *js_policy, kbase_context *current_ctx, kbase_context *new_ctx )
+{
+ kbasep_js_policy_cfs_ctx *current_ctx_info;
+ kbasep_js_policy_cfs_ctx *new_ctx_info;
+
+ OSK_ASSERT( current_ctx != NULL );
+ OSK_ASSERT( new_ctx != NULL );
+ CSTD_UNUSED(js_policy);
+
+ current_ctx_info = ¤t_ctx->jctx.sched_info.runpool.policy_ctx.cfs;
+ new_ctx_info = &new_ctx->jctx.sched_info.runpool.policy_ctx.cfs;
+
+ if((current_ctx_info->process_rt_policy == MALI_FALSE) &&
+ (new_ctx_info->process_rt_policy == MALI_TRUE))
+ {
+ return MALI_TRUE;
+ }
+
+ if((current_ctx_info->process_rt_policy == new_ctx_info->process_rt_policy) &&
+ (current_ctx_info->bag_priority > new_ctx_info->bag_priority))
+ {
+ return MALI_TRUE;
+ }
+
+ return MALI_FALSE;
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_js_policy_cfs.h
+ * Completely Fair Job Scheduler Policy structure definitions
+ */
+
+#ifndef _KBASE_JS_POLICY_CFS_H_
+#define _KBASE_JS_POLICY_CFS_H_
+
+#define KBASE_JS_POLICY_AVAILABLE_CFS
+
+/** @addtogroup base_api
+ * @{ */
+/** @addtogroup base_kbase_api
+ * @{ */
+/** @addtogroup kbase_js_policy
+ * @{ */
+
+/**
+ * Internally, this policy keeps a few internal queues for different variants
+ * of core requirements, which are used to decide how to schedule onto the
+ * different job slots.
+ *
+ * Currently, one extra variant is supported: an NSS variant.
+ *
+ * Must be a power of 2 to keep the lookup math simple
+ */
+#define KBASEP_JS_MAX_NR_CORE_REQ_VARIANTS_LOG2 3
+#define KBASEP_JS_MAX_NR_CORE_REQ_VARIANTS (1u << KBASEP_JS_MAX_NR_CORE_REQ_VARIANTS_LOG2 )
+
+/** Bits needed in the lookup to support all slots */
+#define KBASEP_JS_VARIANT_LOOKUP_BITS_NEEDED (BASE_JM_MAX_NR_SLOTS * KBASEP_JS_MAX_NR_CORE_REQ_VARIANTS)
+/** Number of u32s needed in the lookup array to support all slots */
+#define KBASEP_JS_VARIANT_LOOKUP_WORDS_NEEDED ((KBASEP_JS_VARIANT_LOOKUP_BITS_NEEDED + 31) / 32)
+
+typedef struct kbasep_js_policy_cfs
+{
+ /** List of all contexts in the context queue. Hold
+ * kbasep_js_device_data::queue_mutex whilst accessing. */
+ osk_dlist ctx_queue_head;
+
+ /** List of all contexts in the realtime (priority) context queue */
+ osk_dlist ctx_rt_queue_head;
+
+ /** List of scheduled contexts. Hold kbasep_jd_device_data::runpool_irq::lock
+ * whilst accessing, which is a spinlock */
+ osk_dlist scheduled_ctxs_head;
+
+ /** Number of valid elements in the core_req_variants member, and the
+ * kbasep_js_policy_rr_ctx::job_list_head array */
+ u32 num_core_req_variants;
+
+ /** Variants of the core requirements */
+ kbasep_atom_req core_req_variants[KBASEP_JS_MAX_NR_CORE_REQ_VARIANTS];
+
+ /* Lookups per job slot against which core_req_variants match it */
+ u32 slot_to_variant_lookup_ss_state[KBASEP_JS_VARIANT_LOOKUP_WORDS_NEEDED];
+ u32 slot_to_variant_lookup_ss_allcore_state[KBASEP_JS_VARIANT_LOOKUP_WORDS_NEEDED];
+ u32 slot_to_variant_lookup_nss_state[KBASEP_JS_VARIANT_LOOKUP_WORDS_NEEDED];
+
+ /* The timer tick used for rescheduling jobs */
+ osk_timer timer;
+
+ /* Is the timer running?
+ *
+ * The kbasep_js_device_data::runpool_irq::lock (a spinlock) must be held
+ * whilst accessing this */
+ mali_bool timer_running;
+
+ /* Number of us the least-run context has been running for
+ *
+ * The kbasep_js_device_data::queue_mutex must be held whilst updating this
+ * Reads are possible without this mutex, but an older value might be read
+ * if no memory barriers are issued beforehand. */
+ u64 head_runtime_us;
+} kbasep_js_policy_cfs;
+
+/**
+ * This policy contains a single linked list of all contexts.
+ */
+typedef struct kbasep_js_policy_cfs_ctx
+{
+ /** Link implementing the Policy's Queue, and Currently Scheduled list */
+ osk_dlist_item list;
+
+ /** Job lists for use when in the Run Pool - only using
+ * kbasep_js_policy_fcfs::num_unique_slots of them. We still need to track
+ * the jobs when we're not in the runpool, so this member is accessed from
+ * outside the policy queue (for the first job), inside the policy queue,
+ * and inside the runpool.
+ *
+ * If the context is in the runpool, then this must only be accessed with
+ * kbasep_js_device_data::runpool_irq::lock held
+ *
+ * Jobs are still added to this list even when the context is not in the
+ * runpool. In that case, the kbasep_js_kctx_info::ctx::jsctx_mutex must be
+ * held before accessing this. */
+ osk_dlist job_list_head[KBASEP_JS_MAX_NR_CORE_REQ_VARIANTS];
+
+ /** Number of us this context has been running for
+ *
+ * The kbasep_js_device_data::runpool_irq::lock (a spinlock) must be held
+ * whilst updating this. Initializing will occur on context init and
+ * context enqueue (which can only occur in one thread at a time), but
+ * multi-thread access only occurs while the context is in the runpool.
+ *
+ * Reads are possible without this spinlock, but an older value might be read
+ * if no memory barriers are issued beforehand */
+ u64 runtime_us;
+
+ /* Calling process policy scheme is a realtime scheduler and will use the priority queue
+ * Non-mutable after ctx init */
+ mali_bool process_rt_policy;
+ /* Calling process NICE priority */
+ int process_priority;
+ /* Average NICE priority of all atoms in bag:
+ * Hold the kbasep_js_kctx_info::ctx::jsctx_mutex when accessing */
+ int bag_priority;
+ /* Total NICE priority of all atoms in bag
+ * Hold the kbasep_js_kctx_info::ctx::jsctx_mutex when accessing */
+ int bag_total_priority;
+ /* Total number of atoms in the bag
+ * Hold the kbasep_js_kctx_info::ctx::jsctx_mutex when accessing */
+ int bag_total_nr_atoms;
+
+} kbasep_js_policy_cfs_ctx;
+
+/**
+ * In this policy, each Job is part of at most one of the per_corereq lists
+ */
+typedef struct kbasep_js_policy_cfs_job
+{
+ osk_dlist_item list; /**< Link implementing the Run Pool list/Jobs owned by the ctx */
+ u32 cached_variant_idx; /**< Cached index of the list this should be entered into on re-queue */
+
+ /** Number of ticks that this job has been executing for
+ *
+ * To access this, the kbdev->jm_slots[ js ].lock must be held for the slot 'js'
+ * that that atom is running/queued/about to be queued upon */
+ u32 ticks;
+} kbasep_js_policy_cfs_job;
+
+/** @} */ /* end group kbase_js_policy */
+/** @} */ /* end group base_kbase_api */
+/** @} */ /* end group base_api */
+
+#endif
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_mem.c
+ * Base kernel memory APIs
+ */
+#ifdef CONFIG_DMA_SHARED_BUFFER
+#include <linux/dma-buf.h>
+#endif /* CONFIG_DMA_SHARED_BUFFER */
+#include <osk/mali_osk.h>
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_midg_regmap.h>
+#include <kbase/src/common/mali_kbase_cache_policy.h>
+#include <kbase/src/common/mali_kbase_hw.h>
+#include <kbase/src/common/mali_kbase_gator.h>
+
+typedef struct kbasep_memory_region_performance
+{
+ kbase_memory_performance cpu_performance;
+ kbase_memory_performance gpu_performance;
+} kbasep_memory_region_performance;
+
+static mali_bool kbasep_allocator_order_list_create( osk_phy_allocator * allocators,
+ kbasep_memory_region_performance *region_performance,
+ int memory_region_count, osk_phy_allocator ***sorted_allocs, int allocator_order_count);
+
+/*
+ * An iterator which uses one of the orders listed in kbase_phys_allocator_order enum to iterate over allocators array.
+ */
+typedef struct kbase_phys_allocator_iterator
+{
+ unsigned int cur_idx;
+ kbase_phys_allocator_array * array;
+ kbase_phys_allocator_order order;
+} kbase_phys_allocator_iterator;
+
+
+mali_error kbase_mem_init(kbase_device * kbdev)
+{
+ CSTD_UNUSED(kbdev);
+ /* nothing to do, zero-inited when kbase_device was created */
+ return MALI_ERROR_NONE;
+}
+
+void kbase_mem_halt(kbase_device * kbdev)
+{
+ CSTD_UNUSED(kbdev);
+}
+
+void kbase_mem_term(kbase_device * kbdev)
+{
+ u32 i;
+ kbasep_mem_device * memdev;
+ OSK_ASSERT(kbdev);
+
+ memdev = &kbdev->memdev;
+
+ for (i = 0; i < memdev->allocators.count; i++)
+ {
+ osk_phy_allocator_term(&memdev->allocators.allocs[i]);
+ }
+ osk_free(memdev->allocators.allocs);
+ osk_free(memdev->allocators.sorted_allocs[0]);
+
+ kbase_mem_usage_term(&memdev->usage);
+}
+KBASE_EXPORT_TEST_API(kbase_mem_term)
+
+static mali_error kbase_phys_it_init(kbase_device * kbdev, kbase_phys_allocator_iterator * it, kbase_phys_allocator_order order)
+{
+ OSK_ASSERT(kbdev);
+ OSK_ASSERT(it);
+
+ if (!kbdev->memdev.allocators.count)
+ {
+ return MALI_ERROR_OUT_OF_MEMORY;
+ }
+
+ it->cur_idx = 0;
+ it->array = &kbdev->memdev.allocators;
+ it->order = order;
+
+#if MALI_DEBUG
+ it->array->it_bound = MALI_TRUE;
+#endif /* MALI_DEBUG */
+
+ return MALI_ERROR_NONE;
+}
+
+static void kbase_phys_it_term(kbase_phys_allocator_iterator * it)
+{
+ OSK_ASSERT(it);
+ it->cur_idx = 0;
+#if MALI_DEBUG
+ it->array->it_bound = MALI_FALSE;
+#endif /* MALI_DEBUG */
+ it->array = NULL;
+ return;
+}
+
+static osk_phy_allocator * kbase_phys_it_deref(kbase_phys_allocator_iterator * it)
+{
+ OSK_ASSERT(it);
+ OSK_ASSERT(it->array);
+
+ if (it->cur_idx < it->array->count)
+ {
+ return it->array->sorted_allocs[it->order][it->cur_idx];
+ }
+ else
+ {
+ return NULL;
+ }
+}
+
+static osk_phy_allocator * kbase_phys_it_deref_and_advance(kbase_phys_allocator_iterator * it)
+{
+ osk_phy_allocator * alloc;
+
+ OSK_ASSERT(it);
+ OSK_ASSERT(it->array);
+
+ alloc = kbase_phys_it_deref(it);
+ if (alloc)
+ {
+ it->cur_idx++;
+ }
+ return alloc;
+}
+
+/*
+ * Page free helper.
+ * Handles that commit objects tracks the pages we free
+ */
+static void kbase_free_phy_pages_helper(kbase_va_region * reg, u32 nr_pages);
+
+mali_error kbase_mem_usage_init(kbasep_mem_usage * usage, u32 max_pages)
+{
+ OSK_ASSERT(usage);
+ osk_atomic_set(&usage->cur_pages, 0);
+ /* query the max page count */
+ usage->max_pages = max_pages;
+
+ return MALI_ERROR_NONE;
+}
+
+void kbase_mem_usage_term(kbasep_mem_usage * usage)
+{
+ OSK_ASSERT(usage);
+ /* No memory should be in use now */
+ OSK_ASSERT(0 == osk_atomic_get(&usage->cur_pages));
+ /* So any new alloc requests will fail */
+ usage->max_pages = 0;
+ /* So we assert on double term */
+ osk_atomic_set(&usage->cur_pages, U32_MAX);
+}
+
+mali_error kbase_mem_usage_request_pages(kbasep_mem_usage *usage, u32 nr_pages)
+{
+ u32 cur_pages;
+ u32 old_cur_pages;
+
+ OSK_ASSERT(usage);
+ OSK_ASSERT(nr_pages); /* 0 pages would be an error in the calling code */
+
+ /*
+ * Fetch the initial cur_pages value
+ * each loop iteration below fetches
+ * it as part of the store attempt
+ */
+ cur_pages = osk_atomic_get(&usage->cur_pages);
+
+ /* this check allows the simple if test in the loop below */
+ if (usage->max_pages < nr_pages)
+ {
+ goto usage_cap_exceeded;
+ }
+
+ do
+ {
+ u32 new_cur_pages;
+
+ /* enough pages to fullfill the request? */
+ if (usage->max_pages - nr_pages < cur_pages)
+ {
+usage_cap_exceeded:
+ OSK_PRINT_WARN( OSK_BASE_MEM,
+ "Memory usage cap has been reached:\n"
+ "\t%lu pages currently used\n"
+ "\t%lu pages usage cap\n"
+ "\t%lu new pages requested\n"
+ "\twould result in %lu pages over the cap\n",
+ cur_pages,
+ usage->max_pages,
+ nr_pages,
+ cur_pages + nr_pages - usage->max_pages
+ );
+ return MALI_ERROR_OUT_OF_MEMORY;
+ }
+
+ /* try to atomically commit the new count */
+ old_cur_pages = cur_pages;
+ new_cur_pages = cur_pages + nr_pages;
+ cur_pages = osk_atomic_compare_and_swap(&usage->cur_pages, old_cur_pages, new_cur_pages);
+ /* cur_pages will be like old_cur_pages if there was no race */
+ } while (cur_pages != old_cur_pages);
+
+#if MALI_GATOR_SUPPORT
+ kbase_trace_mali_total_alloc_pages_change((long long int)nr_pages);
+#endif
+
+ return MALI_ERROR_NONE;
+}
+KBASE_EXPORT_TEST_API(kbase_mem_usage_request_pages)
+
+void kbase_mem_usage_release_pages(kbasep_mem_usage * usage, u32 nr_pages)
+{
+ OSK_ASSERT(usage);
+ OSK_ASSERT(nr_pages <= osk_atomic_get(&usage->cur_pages));
+
+ osk_atomic_sub(&usage->cur_pages, nr_pages);
+#if MALI_GATOR_SUPPORT
+ kbase_trace_mali_total_alloc_pages_change(-((long long int)nr_pages));
+#endif
+}
+KBASE_EXPORT_TEST_API(kbase_mem_usage_release_pages)
+
+/**
+ * @brief Wait for GPU write flush - only in use for BASE_HW_ISSUE_6367
+ *
+ * Wait 1000 GPU clock cycles. This delay is known to give the GPU time to flush its write buffer.
+ * @note If GPU resets occur then the counters are reset to zero, the delay may not be as expected.
+ */
+#if MALI_NO_MALI
+static void kbase_wait_write_flush(struct kbase_context *kctx) { }
+#else
+static void kbase_wait_write_flush(struct kbase_context *kctx)
+{
+ u32 base_count = 0;
+ kbase_pm_context_active(kctx->kbdev);
+ kbase_pm_request_gpu_cycle_counter(kctx->kbdev);
+ while( MALI_TRUE )
+ {
+ u32 new_count;
+ new_count = kbase_reg_read(kctx->kbdev, GPU_CONTROL_REG(CYCLE_COUNT_LO), NULL);
+ /* First time around, just store the count. */
+ if( base_count == 0 )
+ {
+ base_count = new_count;
+ continue;
+ }
+
+ /* No need to handle wrapping, unsigned maths works for this. */
+ if( (new_count - base_count) > 1000 )
+ {
+ break;
+ }
+ }
+ kbase_pm_release_gpu_cycle_counter(kctx->kbdev);
+ kbase_pm_context_idle(kctx->kbdev);
+}
+#endif
+
+/**
+ * @brief Check the zone compatibility of two regions.
+ */
+STATIC int kbase_match_zone(struct kbase_va_region *reg1, struct kbase_va_region *reg2)
+{
+ return ((reg1->flags & KBASE_REG_ZONE_MASK) == (reg2->flags & KBASE_REG_ZONE_MASK));
+}
+KBASE_EXPORT_TEST_API(kbase_match_zone)
+
+/**
+ * @brief Allocate a free region object.
+ *
+ * The allocated object is not part of any list yet, and is flagged as
+ * KBASE_REG_FREE. No mapping is allocated yet.
+ *
+ * zone is KBASE_REG_ZONE_TMEM, KBASE_REG_ZONE_PMEM, or KBASE_REG_ZONE_EXEC
+ *
+ */
+struct kbase_va_region *kbase_alloc_free_region(struct kbase_context *kctx, u64 start_pfn, u32 nr_pages, u32 zone)
+{
+ struct kbase_va_region *new_reg;
+
+ OSK_ASSERT(kctx != NULL);
+
+ /* zone argument should only contain zone related region flags */
+ OSK_ASSERT((zone & ~KBASE_REG_ZONE_MASK) == 0);
+ OSK_ASSERT(nr_pages > 0);
+ OSK_ASSERT( start_pfn + nr_pages <= (UINT64_MAX / OSK_PAGE_SIZE) ); /* 64-bit address range is the max */
+
+ new_reg = osk_calloc(sizeof(*new_reg));
+ if (!new_reg)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MEM, "calloc failed");
+ return NULL;
+ }
+
+ new_reg->kctx = kctx;
+ new_reg->flags = zone | KBASE_REG_FREE;
+
+ if ( KBASE_REG_ZONE_TMEM == zone || KBASE_REG_ZONE_EXEC == zone )
+ {
+ new_reg->flags |= KBASE_REG_GROWABLE;
+ }
+
+ /* not imported by default */
+ new_reg->imported_type = BASE_TMEM_IMPORT_TYPE_INVALID;
+
+ new_reg->start_pfn = start_pfn;
+ new_reg->nr_pages = nr_pages;
+ OSK_DLIST_INIT(&new_reg->map_list);
+ new_reg->root_commit.allocator = NULL;
+ new_reg->last_commit = &new_reg->root_commit;
+
+ return new_reg;
+}
+KBASE_EXPORT_TEST_API(kbase_alloc_free_region)
+
+/**
+ * @brief Free a region object.
+ *
+ * The described region must be freed of any mapping.
+ *
+ * If the region is not flagged as KBASE_REG_FREE, the destructor
+ * kbase_free_phy_pages() will be called.
+ */
+void kbase_free_alloced_region(struct kbase_va_region *reg)
+{
+ OSK_ASSERT(NULL != reg);
+ OSK_ASSERT(OSK_DLIST_IS_EMPTY(®->map_list));
+ if (!(reg->flags & KBASE_REG_FREE))
+ {
+ kbase_free_phy_pages(reg);
+ OSK_DEBUG_CODE(
+ /* To detect use-after-free in debug builds */
+ reg->flags |= KBASE_REG_FREE
+ );
+ }
+ osk_free(reg);
+}
+KBASE_EXPORT_TEST_API(kbase_free_alloced_region)
+
+/**
+ * @brief Insert a region object in the global list.
+ *
+ * The region new_reg is inserted at start_pfn by replacing at_reg
+ * partially or completely. at_reg must be a KBASE_REG_FREE region
+ * that contains start_pfn and at least nr_pages from start_pfn. It
+ * must be called with the context region lock held. Internal use
+ * only.
+ */
+static mali_error kbase_insert_va_region_nolock(struct kbase_context *kctx,
+ struct kbase_va_region *new_reg,
+ struct kbase_va_region *at_reg,
+ u64 start_pfn, u32 nr_pages)
+{
+ struct kbase_va_region *new_front_reg;
+ mali_error err = MALI_ERROR_NONE;
+
+ /* Must be a free region */
+ OSK_ASSERT((at_reg->flags & KBASE_REG_FREE) != 0);
+ /* start_pfn should be contained within at_reg */
+ OSK_ASSERT((start_pfn >= at_reg->start_pfn) && (start_pfn < at_reg->start_pfn + at_reg->nr_pages));
+ /* at least nr_pages from start_pfn should be contained within at_reg */
+ OSK_ASSERT(start_pfn + nr_pages <= at_reg->start_pfn + at_reg->nr_pages );
+
+ new_reg->start_pfn = start_pfn;
+ new_reg->nr_pages = nr_pages;
+
+ /* Trivial replacement case */
+ if (at_reg->start_pfn == start_pfn && at_reg->nr_pages == nr_pages)
+ {
+ OSK_DLIST_INSERT_BEFORE(&kctx->reg_list, new_reg, at_reg, struct kbase_va_region, link);
+ OSK_DLIST_REMOVE(&kctx->reg_list, at_reg, link);
+ kbase_free_alloced_region(at_reg);
+ }
+ /* Begin case */
+ else if (at_reg->start_pfn == start_pfn)
+ {
+ at_reg->start_pfn += nr_pages;
+ OSK_ASSERT(at_reg->nr_pages >= nr_pages);
+ at_reg->nr_pages -= nr_pages;
+
+ OSK_DLIST_INSERT_BEFORE(&kctx->reg_list, new_reg, at_reg, struct kbase_va_region, link);
+ }
+ /* End case */
+ else if ((at_reg->start_pfn + at_reg->nr_pages) == (start_pfn + nr_pages))
+ {
+ at_reg->nr_pages -= nr_pages;
+
+ OSK_DLIST_INSERT_AFTER(&kctx->reg_list, new_reg, at_reg, struct kbase_va_region, link);
+ }
+ /* Middle of the road... */
+ else
+ {
+ new_front_reg = kbase_alloc_free_region(kctx, at_reg->start_pfn,
+ start_pfn - at_reg->start_pfn,
+ at_reg->flags & KBASE_REG_ZONE_MASK);
+ if (new_front_reg)
+ {
+ at_reg->nr_pages -= nr_pages + new_front_reg->nr_pages;
+ at_reg->start_pfn = start_pfn + nr_pages;
+
+ OSK_DLIST_INSERT_BEFORE(&kctx->reg_list, new_front_reg, at_reg, struct kbase_va_region, link);
+ OSK_DLIST_INSERT_BEFORE(&kctx->reg_list, new_reg, at_reg, struct kbase_va_region, link);
+ }
+ else
+ {
+ err = MALI_ERROR_OUT_OF_MEMORY;
+ }
+ }
+
+ return err;
+}
+
+/**
+ * @brief Remove a region object from the global list.
+ *
+ * The region reg is removed, possibly by merging with other free and
+ * compatible adjacent regions. It must be called with the context
+ * region lock held. The associated memory is not released (see
+ * kbase_free_alloced_region). Internal use only.
+ */
+STATIC mali_error kbase_remove_va_region(struct kbase_context *kctx, struct kbase_va_region *reg)
+{
+ struct kbase_va_region *prev;
+ struct kbase_va_region *next;
+ int merged_front = 0;
+ int merged_back = 0;
+ mali_error err = MALI_ERROR_NONE;
+
+ prev = OSK_DLIST_PREV(reg, struct kbase_va_region, link);
+ if (!OSK_DLIST_IS_VALID(prev, link))
+ {
+ prev = NULL;
+ }
+
+ next = OSK_DLIST_NEXT(reg, struct kbase_va_region, link);
+ OSK_ASSERT(NULL != next);
+ if (!OSK_DLIST_IS_VALID(next, link))
+ {
+ next = NULL;
+ }
+
+ /* Try to merge with front first */
+ if (prev && (prev->flags & KBASE_REG_FREE) && kbase_match_zone(prev, reg))
+ {
+ /* We're compatible with the previous VMA, merge with it */
+ OSK_DLIST_REMOVE(&kctx->reg_list, reg, link);
+ prev->nr_pages += reg->nr_pages;
+ reg = prev;
+ merged_front = 1;
+ }
+
+ /* Try to merge with back next */
+ if (next && (next->flags & KBASE_REG_FREE) && kbase_match_zone(next, reg))
+ {
+ /* We're compatible with the next VMA, merge with it */
+ next->start_pfn = reg->start_pfn;
+ next->nr_pages += reg->nr_pages;
+ OSK_DLIST_REMOVE(&kctx->reg_list, reg, link);
+
+ if (merged_front)
+ {
+ /* we already merged with prev, free it */
+ kbase_free_alloced_region(prev);
+ }
+
+ merged_back = 1;
+ }
+
+ if (!(merged_front || merged_back))
+ {
+ /*
+ * We didn't merge anything. Add a new free
+ * placeholder and remove the original one.
+ */
+ struct kbase_va_region *free_reg;
+
+ free_reg = kbase_alloc_free_region(kctx, reg->start_pfn, reg->nr_pages, reg->flags & KBASE_REG_ZONE_MASK);
+ if (!free_reg)
+ {
+ err = MALI_ERROR_OUT_OF_MEMORY;
+ goto out;
+ }
+
+ OSK_DLIST_INSERT_BEFORE(&kctx->reg_list, free_reg, reg, struct kbase_va_region, link);
+ OSK_DLIST_REMOVE(&kctx->reg_list, reg, link);
+ }
+
+out:
+ return err;
+}
+KBASE_EXPORT_TEST_API(kbase_remove_va_region)
+
+/**
+ * @brief Add a region to the global list.
+ *
+ * Add reg to the global list, according to its zone. If addr is
+ * non-null, this address is used directly (as in the PMEM
+ * case). Alignment can be enforced by specifying a number of pages
+ * (which *must* be a power of 2).
+ *
+ * Context region list lock must be held.
+ *
+ * Mostly used by kbase_gpu_mmap(), but also useful to register the
+ * ring-buffer region.
+ */
+mali_error kbase_add_va_region(struct kbase_context *kctx,
+ struct kbase_va_region *reg,
+ mali_addr64 addr, u32 nr_pages,
+ u32 align)
+{
+ struct kbase_va_region *tmp;
+ u64 gpu_pfn = addr >> OSK_PAGE_SHIFT;
+ mali_error err = MALI_ERROR_NONE;
+
+ OSK_ASSERT(NULL != kctx);
+ OSK_ASSERT(NULL != reg);
+
+ if (!align)
+ {
+ align = 1;
+ }
+
+ /* must be a power of 2 */
+ OSK_ASSERT((align & (align - 1)) == 0);
+ OSK_ASSERT( nr_pages > 0 );
+
+ if (gpu_pfn)
+ {
+ OSK_ASSERT(!(gpu_pfn & (align - 1)));
+
+ /*
+ * So we want a specific address. Parse the list until
+ * we find the enclosing region, which *must* be free.
+ */
+ OSK_DLIST_FOREACH(&kctx->reg_list, struct kbase_va_region, link, tmp)
+ {
+ if (tmp->start_pfn <= gpu_pfn &&
+ (tmp->start_pfn + tmp->nr_pages) >= (gpu_pfn + nr_pages))
+ {
+ /* We have the candidate */
+ if (!kbase_match_zone(tmp, reg))
+ {
+ /* Wrong zone, fail */
+ err = MALI_ERROR_OUT_OF_GPU_MEMORY;
+ OSK_PRINT_WARN(OSK_BASE_MEM, "Zone mismatch: %d != %d", tmp->flags & KBASE_REG_ZONE_MASK, reg->flags & KBASE_REG_ZONE_MASK);
+ goto out;
+ }
+
+ if (!(tmp->flags & KBASE_REG_FREE))
+ {
+ OSK_PRINT_WARN(OSK_BASE_MEM, "!(tmp->flags & KBASE_REG_FREE): tmp->start_pfn=0x%llx tmp->flags=0x%x tmp->nr_pages=0x%x gpu_pfn=0x%llx nr_pages=0x%x\n",
+ tmp->start_pfn, tmp->flags, tmp->nr_pages, gpu_pfn, nr_pages);
+ OSK_PRINT_WARN(OSK_BASE_MEM, "in function %s (%p, %p, 0x%llx, 0x%x, 0x%x)\n", __func__,
+ kctx,reg,addr, nr_pages, align);
+ /* Busy, fail */
+ err = MALI_ERROR_OUT_OF_GPU_MEMORY;
+ goto out;
+ }
+
+ err = kbase_insert_va_region_nolock(kctx, reg, tmp, gpu_pfn, nr_pages);
+ if (err) OSK_PRINT_WARN(OSK_BASE_MEM, "Failed to insert va region");
+ goto out;
+ }
+ }
+
+ err = MALI_ERROR_OUT_OF_GPU_MEMORY;
+ OSK_PRINT_WARN(OSK_BASE_MEM, "Out of mem");
+ goto out;
+ }
+
+ /* Find the first free region that accomodates our requirements */
+ OSK_DLIST_FOREACH(&kctx->reg_list, struct kbase_va_region, link, tmp)
+ {
+ if (tmp->nr_pages >= nr_pages &&
+ (tmp->flags & KBASE_REG_FREE) &&
+ kbase_match_zone(tmp, reg))
+ {
+ /* Check alignment */
+ u64 start_pfn;
+ start_pfn = (tmp->start_pfn + align - 1) & ~(align - 1);
+
+ if (
+ (start_pfn >= tmp->start_pfn) &&
+ (start_pfn <= (tmp->start_pfn + (tmp->nr_pages - 1))) &&
+ ((start_pfn + nr_pages - 1) <= (tmp->start_pfn + tmp->nr_pages -1))
+ )
+ {
+ /* It fits, let's use it */
+ err = kbase_insert_va_region_nolock(kctx, reg, tmp, start_pfn, nr_pages);
+ goto out;
+ }
+ }
+ }
+
+ err = MALI_ERROR_OUT_OF_GPU_MEMORY;
+out:
+ return err;
+}
+KBASE_EXPORT_TEST_API(kbase_add_va_region)
+
+void kbase_mmu_update(struct kbase_context *kctx)
+{
+ /* Use GPU implementation-defined caching policy. */
+ u32 memattr = ASn_MEMATTR_IMPL_DEF_CACHE_POLICY;
+ u32 pgd_high;
+
+ OSK_ASSERT(NULL != kctx);
+ /* ASSERT that the context has a valid as_nr, which is only the case
+ * when it's scheduled in.
+ *
+ * as_nr won't change because the caller has the runpool_irq lock */
+ OSK_ASSERT( kctx->as_nr != KBASEP_AS_NR_INVALID );
+
+ pgd_high = sizeof(kctx->pgd)>4?(kctx->pgd >> 32):0;
+
+ kbase_reg_write(kctx->kbdev,
+ MMU_AS_REG(kctx->as_nr, ASn_TRANSTAB_LO),
+ (kctx->pgd & ASn_TRANSTAB_ADDR_SPACE_MASK) | ASn_TRANSTAB_READ_INNER
+ | ASn_TRANSTAB_ADRMODE_TABLE, kctx);
+
+ /* Need to use a conditional expression to avoid "right shift count >= width of type"
+ * error when using an if statement - although the size_of condition is evaluated at compile
+ * time the unused branch is not removed until after it is type-checked and the error
+ * produced.
+ */
+ pgd_high = sizeof(kctx->pgd)>4?(kctx->pgd >> 32):0;
+
+ kbase_reg_write(kctx->kbdev,
+ MMU_AS_REG(kctx->as_nr, ASn_TRANSTAB_HI),
+ pgd_high, kctx);
+
+ kbase_reg_write(kctx->kbdev,
+ MMU_AS_REG(kctx->as_nr, ASn_MEMATTR_LO),
+ memattr, kctx);
+ kbase_reg_write(kctx->kbdev,
+ MMU_AS_REG(kctx->as_nr, ASn_MEMATTR_HI),
+ memattr, kctx);
+ kbase_reg_write(kctx->kbdev,
+ MMU_AS_REG(kctx->as_nr, ASn_COMMAND),
+ ASn_COMMAND_UPDATE, kctx);
+}
+KBASE_EXPORT_TEST_API(kbase_mmu_update)
+
+void kbase_mmu_disable (kbase_context *kctx)
+{
+ OSK_ASSERT(NULL != kctx);
+ /* ASSERT that the context has a valid as_nr, which is only the case
+ * when it's scheduled in.
+ *
+ * as_nr won't change because the caller has the runpool_irq lock */
+ OSK_ASSERT( kctx->as_nr != KBASEP_AS_NR_INVALID );
+
+ kbase_reg_write(kctx->kbdev,
+ MMU_AS_REG(kctx->as_nr, ASn_TRANSTAB_LO),
+ 0, kctx);
+ kbase_reg_write(kctx->kbdev,
+ MMU_AS_REG(kctx->as_nr, ASn_TRANSTAB_HI),
+ 0, kctx);
+ kbase_reg_write(kctx->kbdev,
+ MMU_AS_REG(kctx->as_nr, ASn_COMMAND),
+ ASn_COMMAND_UPDATE, kctx);
+}
+KBASE_EXPORT_TEST_API(kbase_mmu_disable)
+
+mali_error kbase_gpu_mmap(struct kbase_context *kctx,
+ struct kbase_va_region *reg,
+ mali_addr64 addr, u32 nr_pages,
+ u32 align)
+{
+ mali_error err;
+ OSK_ASSERT(NULL != kctx);
+ OSK_ASSERT(NULL != reg);
+
+ err = kbase_add_va_region(kctx, reg, addr, nr_pages, align);
+ if (MALI_ERROR_NONE != err)
+ {
+ return err;
+ }
+
+ err = kbase_mmu_insert_pages(kctx, reg->start_pfn,
+ kbase_get_phy_pages(reg),
+ reg->nr_alloc_pages, reg->flags & ((1 << KBASE_REG_FLAGS_NR_BITS)-1));
+ if(MALI_ERROR_NONE != err)
+ {
+ kbase_remove_va_region(kctx, reg);
+ }
+
+ return err;
+}
+KBASE_EXPORT_TEST_API(kbase_gpu_mmap)
+
+mali_error kbase_gpu_munmap(struct kbase_context *kctx, struct kbase_va_region *reg)
+{
+ mali_error err;
+
+ if(reg->start_pfn == 0 )
+ {
+ return MALI_ERROR_NONE;
+ }
+ err = kbase_mmu_teardown_pages(kctx, reg->start_pfn, reg->nr_alloc_pages);
+ if(MALI_ERROR_NONE != err)
+ {
+ return err;
+ }
+
+ err = kbase_remove_va_region(kctx, reg);
+ return err;
+}
+
+kbase_va_region *kbase_region_lookup(kbase_context *kctx, mali_addr64 gpu_addr)
+{
+ kbase_va_region *tmp;
+ u64 gpu_pfn = gpu_addr >> OSK_PAGE_SHIFT;
+ OSK_ASSERT(NULL != kctx);
+
+ OSK_DLIST_FOREACH(&kctx->reg_list, kbase_va_region, link, tmp)
+ {
+ if (gpu_pfn >= tmp->start_pfn && (gpu_pfn < tmp->start_pfn + tmp->nr_pages))
+ {
+ return tmp;
+ }
+ }
+
+ return NULL;
+}
+KBASE_EXPORT_TEST_API(kbase_region_lookup)
+
+struct kbase_va_region *kbase_validate_region(struct kbase_context *kctx, mali_addr64 gpu_addr)
+{
+ struct kbase_va_region *tmp;
+ u64 gpu_pfn = gpu_addr >> OSK_PAGE_SHIFT;
+ OSK_ASSERT(NULL != kctx);
+
+ OSK_DLIST_FOREACH(&kctx->reg_list, struct kbase_va_region, link, tmp)
+ {
+ if (tmp->start_pfn == gpu_pfn)
+ {
+ return tmp;
+ }
+ }
+
+ return NULL;
+}
+KBASE_EXPORT_TEST_API(kbase_validate_region)
+
+/**
+ * @brief Find a mapping keyed with ptr in region reg
+ */
+STATIC struct kbase_cpu_mapping *kbase_find_cpu_mapping(struct kbase_va_region *reg,
+ const void *ptr)
+{
+ struct kbase_cpu_mapping *map;
+ OSK_ASSERT(NULL != reg);
+ OSK_DLIST_FOREACH(®->map_list, struct kbase_cpu_mapping, link, map)
+ {
+ if (map->private == ptr)
+ {
+ return map;
+ }
+ }
+
+ return NULL;
+}
+KBASE_EXPORT_TEST_API(kbase_find_cpu_mapping)
+
+STATIC struct kbase_cpu_mapping *kbasep_find_enclosing_cpu_mapping_of_region(
+ const struct kbase_va_region *reg,
+ osk_virt_addr uaddr,
+ size_t size)
+{
+ struct kbase_cpu_mapping *map;
+
+ OSK_ASSERT(NULL != reg);
+
+ if ((uintptr_t)uaddr + size < (uintptr_t)uaddr) /* overflow check */
+ {
+ return NULL;
+ }
+
+ OSK_DLIST_FOREACH(®->map_list, struct kbase_cpu_mapping, link, map)
+ {
+ if (map->uaddr <= uaddr &&
+ ((uintptr_t)map->uaddr + (map->nr_pages << OSK_PAGE_SHIFT)) >= ((uintptr_t)uaddr + size))
+ {
+ return map;
+ }
+ }
+
+ return NULL;
+}
+KBASE_EXPORT_TEST_API(kbasep_find_enclosing_cpu_mapping_of_region)
+
+static void kbase_dump_mappings(struct kbase_va_region *reg)
+{
+ struct kbase_cpu_mapping *map;
+
+ OSK_ASSERT(NULL != reg);
+
+ OSK_DLIST_FOREACH(®->map_list, struct kbase_cpu_mapping, link, map)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MEM, "uaddr %p nr_pages %d page_off %016llx vma %p\n",
+ map->uaddr, map->nr_pages,
+ map->page_off, map->private);
+ }
+}
+
+/**
+ * @brief Delete a mapping keyed with ptr in region reg
+ */
+mali_error kbase_cpu_free_mapping(struct kbase_va_region *reg, const void *ptr)
+{
+ struct kbase_cpu_mapping *map;
+ mali_error err = MALI_ERROR_NONE;
+ OSK_ASSERT(NULL != reg);
+ map = kbase_find_cpu_mapping(reg, ptr);
+ if (!map)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MEM, "Freeing unknown mapping %p in region %p\n", ptr, (void*)reg);
+ kbase_dump_mappings(reg);
+ err = MALI_ERROR_FUNCTION_FAILED;
+ goto out;
+ }
+
+ OSK_DLIST_REMOVE(®->map_list, map, link);
+ osk_free(map);
+out:
+ return err;
+}
+KBASE_EXPORT_TEST_API(kbase_cpu_free_mapping)
+
+struct kbase_cpu_mapping *kbasep_find_enclosing_cpu_mapping(
+ struct kbase_context *kctx,
+ mali_addr64 gpu_addr,
+ osk_virt_addr uaddr,
+ size_t size )
+{
+ struct kbase_cpu_mapping *map = NULL;
+ const struct kbase_va_region *reg;
+
+ OSKP_ASSERT( kctx != NULL );
+
+ kbase_gpu_vm_lock(kctx);
+
+ reg = kbase_region_lookup( kctx, gpu_addr );
+ if ( NULL != reg )
+ {
+ map = kbasep_find_enclosing_cpu_mapping_of_region( reg, uaddr, size);
+ }
+
+ kbase_gpu_vm_unlock(kctx);
+
+ return map;
+}
+KBASE_EXPORT_TEST_API(kbasep_find_enclosing_cpu_mapping)
+
+static mali_error kbase_do_syncset(struct kbase_context *kctx, base_syncset *set,
+ osk_sync_kmem_fn sync_fn)
+{
+ mali_error err = MALI_ERROR_NONE;
+ struct basep_syncset *sset = &set->basep_sset;
+ struct kbase_va_region *reg;
+ struct kbase_cpu_mapping *map;
+ osk_phy_addr *pa;
+ u64 page_off, page_count, size_in_pages;
+ osk_virt_addr start;
+ size_t size;
+ u64 i;
+ u32 offset_within_page;
+ osk_phy_addr base_phy_addr = 0;
+ osk_virt_addr base_virt_addr = 0;
+ size_t area_size = 0;
+
+ kbase_os_mem_map_lock(kctx);
+
+ kbase_gpu_vm_lock(kctx);
+
+ /* find the region where the virtual address is contained */
+ reg = kbase_region_lookup(kctx, sset->mem_handle);
+ if (!reg)
+ {
+ err = MALI_ERROR_FUNCTION_FAILED;
+ goto out_unlock;
+ }
+
+ if (!(reg->flags & KBASE_REG_CPU_CACHED))
+ {
+ goto out_unlock;
+ }
+
+ start = (osk_virt_addr)(uintptr_t)sset->user_addr;
+ size = sset->size;
+
+ map = kbasep_find_enclosing_cpu_mapping_of_region(reg, start, size);
+ if (!map)
+ {
+ err = MALI_ERROR_FUNCTION_FAILED;
+ goto out_unlock;
+ }
+
+ offset_within_page = (uintptr_t)start & (OSK_PAGE_SIZE - 1);
+ size_in_pages = (size + offset_within_page + (OSK_PAGE_SIZE - 1)) & OSK_PAGE_MASK;
+ page_off = map->page_off + (((uintptr_t)start - (uintptr_t)map->uaddr) >> OSK_PAGE_SHIFT);
+ page_count = (size_in_pages >> OSK_PAGE_SHIFT);
+ pa = kbase_get_phy_pages(reg);
+
+ for (i = 0; i < page_count; i++)
+ {
+ u32 offset = (uintptr_t)start & (OSK_PAGE_SIZE - 1);
+ osk_phy_addr paddr = pa[page_off + i] + offset;
+ size_t sz = OSK_MIN(((size_t)OSK_PAGE_SIZE - offset), size);
+
+ if (paddr == base_phy_addr + area_size &&
+ start == (osk_virt_addr)((uintptr_t)base_virt_addr + area_size))
+ {
+ area_size += sz;
+ }
+ else if (area_size > 0)
+ {
+ sync_fn(base_phy_addr, base_virt_addr, area_size);
+ area_size = 0;
+ }
+
+ if (area_size == 0)
+ {
+ base_phy_addr = paddr;
+ base_virt_addr = start;
+ area_size = sz;
+ }
+
+ start = (osk_virt_addr)((uintptr_t)start + sz);
+ size -= sz;
+ }
+
+ if (area_size > 0)
+ {
+ sync_fn(base_phy_addr, base_virt_addr, area_size);
+ }
+
+ OSK_ASSERT(size == 0);
+
+out_unlock:
+ kbase_gpu_vm_unlock(kctx);
+ kbase_os_mem_map_unlock(kctx);
+ return err;
+}
+
+static mali_error kbase_sync_to_memory(kbase_context *kctx, base_syncset *syncset)
+{
+ return kbase_do_syncset(kctx, syncset, osk_sync_to_memory);
+}
+
+static mali_error kbase_sync_to_cpu(kbase_context *kctx, base_syncset *syncset)
+{
+ return kbase_do_syncset(kctx, syncset, osk_sync_to_cpu);
+}
+
+mali_error kbase_sync_now(kbase_context *kctx, base_syncset *syncset)
+{
+ mali_error err = MALI_ERROR_FUNCTION_FAILED;
+ struct basep_syncset *sset;
+
+ OSK_ASSERT( NULL != kctx );
+ OSK_ASSERT( NULL != syncset );
+
+ sset = &syncset->basep_sset;
+
+ switch(sset->type)
+ {
+ case BASE_SYNCSET_OP_MSYNC:
+ err = kbase_sync_to_memory(kctx, syncset);
+ break;
+
+ case BASE_SYNCSET_OP_CSYNC:
+ err = kbase_sync_to_cpu(kctx, syncset);
+ break;
+
+ default:
+ OSK_PRINT_WARN(OSK_BASE_MEM, "Unknown msync op %d\n", sset->type);
+ break;
+ }
+
+ return err;
+}
+KBASE_EXPORT_TEST_API(kbase_sync_now)
+
+void kbase_pre_job_sync(kbase_context *kctx, base_syncset *syncsets, u32 nr)
+{
+ u32 i;
+
+ OSK_ASSERT(NULL != kctx);
+ OSK_ASSERT(NULL != syncsets);
+
+ for (i = 0; i < nr; i++)
+ {
+ u8 type = syncsets[i].basep_sset.type;
+
+ switch(type)
+ {
+ case BASE_SYNCSET_OP_MSYNC:
+ kbase_sync_to_memory(kctx, &syncsets[i]);
+ break;
+
+ case BASE_SYNCSET_OP_CSYNC:
+ continue;
+
+ default:
+ OSK_PRINT_WARN(OSK_BASE_MEM, "Unknown msync op %d\n", type);
+ break;
+ }
+ }
+}
+KBASE_EXPORT_TEST_API(kbase_pre_job_sync)
+
+void kbase_post_job_sync(kbase_context *kctx, base_syncset *syncsets, u32 nr)
+{
+ u32 i;
+
+ OSK_ASSERT(NULL != kctx);
+ OSK_ASSERT(NULL != syncsets);
+
+ for (i = 0; i < nr; i++)
+ {
+ struct basep_syncset *sset = &syncsets[i].basep_sset;
+ switch(sset->type)
+ {
+ case BASE_SYNCSET_OP_CSYNC:
+ kbase_sync_to_cpu(kctx, &syncsets[i]);
+ break;
+
+ case BASE_SYNCSET_OP_MSYNC:
+ continue;
+
+ default:
+ OSK_PRINT_WARN(OSK_BASE_MEM, "Unknown msync op %d\n", sset->type);
+ break;
+ }
+ }
+}
+KBASE_EXPORT_TEST_API(kbase_post_job_sync)
+
+/* vm lock must be held */
+mali_error kbase_mem_free_region(struct kbase_context *kctx,
+ kbase_va_region *reg)
+{
+ mali_error err;
+ OSK_ASSERT(NULL != kctx);
+ OSK_ASSERT(NULL != reg);
+
+ if (!OSK_DLIST_IS_EMPTY(®->map_list))
+ {
+ /*
+ * We still have mappings, can't free
+ * memory. This also handles the race
+ * condition with the unmap code (see
+ * kbase_cpu_vm_close()).
+ */
+ OSK_PRINT_WARN(OSK_BASE_MEM, "Pending CPU mappings, not freeing memory!\n");
+ err = MALI_ERROR_FUNCTION_FAILED;
+ goto out;
+ }
+
+ err = kbase_gpu_munmap(kctx, reg);
+ if (err)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MEM, "Could not unmap from the GPU...\n");
+ goto out;
+ }
+
+ if (kbase_hw_has_issue(kctx->kbdev, BASE_HW_ISSUE_6367))
+ {
+ /* Wait for GPU to flush write buffer before freeing physical pages */
+ kbase_wait_write_flush(kctx);
+ }
+
+ /* This will also free the physical pages */
+ kbase_free_alloced_region(reg);
+
+out:
+ return err;
+}
+KBASE_EXPORT_TEST_API(kbase_mem_free_region)
+
+/**
+ * @brief Free the region from the GPU and unregister it.
+ *
+ * This function implements the free operation on a memory segment.
+ * It will loudly fail if called with outstanding mappings.
+ */
+mali_error kbase_mem_free(struct kbase_context *kctx, mali_addr64 gpu_addr)
+{
+ mali_error err = MALI_ERROR_NONE;
+ struct kbase_va_region *reg;
+
+ OSK_ASSERT(kctx != NULL);
+
+ if (0 == gpu_addr)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MEM, "gpu_addr 0 is reserved for the ringbuffer and it's an error to try to free it using kbase_mem_free\n");
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+ kbase_gpu_vm_lock(kctx);
+
+ if (gpu_addr < OSK_PAGE_SIZE)
+ {
+ /* an OS specific cookie, ask the OS specific code to validate it */
+ reg = kbase_lookup_cookie(kctx, gpu_addr);
+ if (!reg)
+ {
+ err = MALI_ERROR_FUNCTION_FAILED;
+ goto out_unlock;
+ }
+
+ /* ask to unlink the cookie as we'll free it */
+ kbase_unlink_cookie(kctx, gpu_addr, reg);
+
+ kbase_free_alloced_region(reg);
+ }
+ else
+ {
+ /* A real GPU va */
+
+ /* Validate the region */
+ reg = kbase_validate_region(kctx, gpu_addr);
+ if (!reg)
+ {
+ OSK_ASSERT_MSG(0, "Trying to free nonexistent region\n 0x%llX", gpu_addr);
+ err = MALI_ERROR_FUNCTION_FAILED;
+ goto out_unlock;
+ }
+
+ err = kbase_mem_free_region(kctx, reg);
+ }
+
+out_unlock:
+ kbase_gpu_vm_unlock(kctx);
+ return err;
+}
+KBASE_EXPORT_TEST_API(kbase_mem_free)
+
+void kbase_update_region_flags(struct kbase_va_region *reg, u32 flags, mali_bool is_growable)
+{
+ OSK_ASSERT(NULL != reg);
+ OSK_ASSERT((flags & ~((1 << BASE_MEM_FLAGS_NR_BITS) - 1)) == 0);
+
+ reg->flags |= kbase_cache_enabled(flags, reg->nr_pages);
+
+ if ((flags & BASE_MEM_GROW_ON_GPF) || is_growable)
+ {
+ reg->flags |= KBASE_REG_GROWABLE;
+
+ if (flags & BASE_MEM_GROW_ON_GPF)
+ {
+ reg->flags |= KBASE_REG_PF_GROW;
+ }
+ }
+ else
+ {
+ /* As this region is not growable but the default is growable, we
+ explicitly clear the growable flag. */
+ reg->flags &= ~KBASE_REG_GROWABLE;
+ }
+
+ if (flags & BASE_MEM_PROT_CPU_WR)
+ {
+ reg->flags |= KBASE_REG_CPU_WR;
+ }
+
+ if (flags & BASE_MEM_PROT_CPU_RD)
+ {
+ reg->flags |= KBASE_REG_CPU_RD;
+ }
+
+ if (flags & BASE_MEM_PROT_GPU_WR)
+ {
+ reg->flags |= KBASE_REG_GPU_WR;
+ }
+
+ if (flags & BASE_MEM_PROT_GPU_RD)
+ {
+ reg->flags |= KBASE_REG_GPU_RD;
+ }
+
+ if (0 == (flags & BASE_MEM_PROT_GPU_EX))
+ {
+ reg->flags |= KBASE_REG_GPU_NX;
+ }
+
+ if (flags & BASE_MEM_COHERENT_LOCAL)
+ {
+ reg->flags |= KBASE_REG_SHARE_IN;
+ }
+ else if (flags & BASE_MEM_COHERENT_SYSTEM)
+ {
+ reg->flags |= KBASE_REG_SHARE_BOTH;
+ }
+}
+
+static void kbase_free_phy_pages_helper(kbase_va_region * reg, u32 nr_pages_to_free)
+{
+ osk_phy_addr *page_array;
+
+ u32 nr_pages;
+
+ OSK_ASSERT(reg);
+ OSK_ASSERT(reg->kctx);
+
+ /* Can't call this on TB buffers */
+ OSK_ASSERT(0 == (reg->flags & KBASE_REG_IS_TB));
+ /* can't be called on imported types */
+ OSK_ASSERT(BASE_TMEM_IMPORT_TYPE_INVALID == reg->imported_type);
+ /* Free of too many pages attempted! */
+ OSK_ASSERT(reg->nr_alloc_pages >= nr_pages_to_free);
+ /* A complete free is required if not marked as growable */
+ OSK_ASSERT((reg->flags & KBASE_REG_GROWABLE) || (reg->nr_alloc_pages == nr_pages_to_free));
+
+ if (0 == nr_pages_to_free)
+ {
+ /* early out if nothing to free */
+ return;
+ }
+
+ nr_pages = nr_pages_to_free;
+
+ page_array = kbase_get_phy_pages(reg);
+
+ OSK_ASSERT(nr_pages_to_free == 0 || page_array != NULL);
+
+ while (nr_pages)
+ {
+ kbase_mem_commit * commit;
+ commit = reg->last_commit;
+
+ if (nr_pages >= commit->nr_pages)
+ {
+ /* free the whole commit */
+ kbase_phy_pages_free(reg->kctx->kbdev, commit->allocator, commit->nr_pages,
+ page_array + reg->nr_alloc_pages - commit->nr_pages);
+
+ /* update page counts */
+ nr_pages -= commit->nr_pages;
+ reg->nr_alloc_pages -= commit->nr_pages;
+
+ /* free the node (unless it's the root node) */
+ if (commit != ®->root_commit)
+ {
+ reg->last_commit = commit->prev;
+ osk_free(commit);
+ }
+ else
+ {
+ /* mark the root node as having no commit */
+ commit->nr_pages = 0;
+ OSK_ASSERT(nr_pages == 0);
+ OSK_ASSERT(reg->nr_alloc_pages == 0);
+ break;
+ }
+ }
+ else
+ {
+ /* partial free of this commit */
+ kbase_phy_pages_free(reg->kctx->kbdev, commit->allocator, nr_pages,
+ page_array + reg->nr_alloc_pages - nr_pages);
+ commit->nr_pages -= nr_pages;
+ reg->nr_alloc_pages -= nr_pages;
+ break; /* end the loop */
+ }
+ }
+
+ kbase_mem_usage_release_pages(®->kctx->usage, nr_pages_to_free);
+}
+KBASE_EXPORT_TEST_API(kbase_update_region_flags)
+
+u32 kbase_phy_pages_alloc(struct kbase_device *kbdev, osk_phy_allocator *allocator, u32 nr_pages,
+ osk_phy_addr *pages)
+{
+ OSK_ASSERT(kbdev != NULL);
+ OSK_ASSERT(allocator != NULL);
+ OSK_ASSERT(pages != NULL);
+
+ if (allocator->type == OSKP_PHY_ALLOCATOR_OS)
+ {
+ u32 pages_allocated;
+
+ /* Claim pages from OS shared quota. Note that shared OS memory may be used by different allocators. That's why
+ * page request is made here and not on per-allocator basis */
+ if (MALI_ERROR_NONE != kbase_mem_usage_request_pages(&kbdev->memdev.usage, nr_pages))
+ {
+ return 0;
+ }
+
+ pages_allocated = osk_phy_pages_alloc(allocator, nr_pages, pages);
+
+ if (pages_allocated < nr_pages)
+ {
+ kbase_mem_usage_release_pages(&kbdev->memdev.usage, nr_pages - pages_allocated);
+ }
+ return pages_allocated;
+ }
+ else
+ {
+ /* Dedicated memory is tracked per allocator. Memory limits are checked in osk_phy_pages_alloc function */
+ return osk_phy_pages_alloc(allocator, nr_pages, pages);
+ }
+}
+KBASE_EXPORT_TEST_API(kbase_phy_pages_alloc)
+
+void kbase_phy_pages_free(struct kbase_device *kbdev, osk_phy_allocator *allocator, u32 nr_pages, osk_phy_addr *pages)
+{
+ OSK_ASSERT(kbdev != NULL);
+ OSK_ASSERT(allocator != NULL);
+ OSK_ASSERT(pages != NULL);
+
+ osk_phy_pages_free(allocator, nr_pages, pages);
+
+ if (allocator->type == OSKP_PHY_ALLOCATOR_OS)
+ {
+ /* release pages from OS shared quota */
+ kbase_mem_usage_release_pages(&kbdev->memdev.usage, nr_pages);
+ }
+}
+KBASE_EXPORT_TEST_API(kbase_phy_pages_free)
+
+
+mali_error kbase_alloc_phy_pages_helper(struct kbase_va_region *reg, u32 nr_pages_requested)
+{
+ kbase_phys_allocator_iterator it;
+ osk_phy_addr *page_array;
+ u32 nr_pages_left;
+ u32 num_pages_on_start;
+ u32 pages_committed;
+ kbase_phys_allocator_order order;
+ u32 performance_flags;
+
+ OSK_ASSERT(reg);
+ OSK_ASSERT(reg->kctx);
+
+ /* Can't call this on TB or UMP buffers */
+ OSK_ASSERT(0 == (reg->flags & KBASE_REG_IS_TB));
+ /* can't be called on imported types */
+ OSK_ASSERT(BASE_TMEM_IMPORT_TYPE_INVALID == reg->imported_type);
+ /* Growth of too many pages attempted! (written this way to catch overflow)) */
+ OSK_ASSERT(reg->nr_pages - reg->nr_alloc_pages >= nr_pages_requested);
+ /* A complete commit is required if not marked as growable */
+ OSK_ASSERT((reg->flags & KBASE_REG_GROWABLE) || (reg->nr_pages == nr_pages_requested));
+
+ if (0 == nr_pages_requested)
+ {
+ /* early out if nothing to do */
+ return MALI_ERROR_NONE;
+ }
+
+ /* track the number pages so we can roll back on alloc fail */
+ num_pages_on_start = reg->nr_alloc_pages;
+ nr_pages_left = nr_pages_requested;
+
+ page_array = kbase_get_phy_pages(reg);
+ OSK_ASSERT(page_array);
+
+ /* claim the pages from our per-context quota */
+ if (MALI_ERROR_NONE != kbase_mem_usage_request_pages(®->kctx->usage, nr_pages_requested))
+ {
+ return MALI_ERROR_OUT_OF_MEMORY;
+ }
+
+ /* First try to extend the last commit */
+ if (reg->last_commit->allocator)
+ {
+ pages_committed = kbase_phy_pages_alloc(reg->kctx->kbdev, reg->last_commit->allocator, nr_pages_left,
+ page_array + reg->nr_alloc_pages);
+ reg->last_commit->nr_pages += pages_committed;
+ reg->nr_alloc_pages += pages_committed;
+ nr_pages_left -= pages_committed;
+
+ if (!nr_pages_left)
+ {
+ return MALI_ERROR_NONE;
+ }
+ }
+
+ performance_flags = reg->flags & (KBASE_REG_CPU_CACHED | KBASE_REG_GPU_CACHED);
+
+ if (performance_flags == 0)
+ {
+ order = ALLOCATOR_ORDER_CONFIG;
+ }
+ else if (performance_flags == KBASE_REG_CPU_CACHED)
+ {
+ order = ALLOCATOR_ORDER_CPU_PERFORMANCE;
+ }
+ else if (performance_flags == KBASE_REG_GPU_CACHED)
+ {
+ order = ALLOCATOR_ORDER_GPU_PERFORMANCE;
+ }
+ else
+ {
+ order = ALLOCATOR_ORDER_CPU_GPU_PERFORMANCE;
+ }
+
+ /* If not fully commited (or no prev allocator) we need to ask all the allocators */
+
+ /* initialize the iterator we use to loop over the memory providers */
+ if (MALI_ERROR_NONE == kbase_phys_it_init(reg->kctx->kbdev, &it, order))
+ {
+ for (;nr_pages_left && kbase_phys_it_deref(&it); kbase_phys_it_deref_and_advance(&it))
+ {
+ pages_committed = kbase_phy_pages_alloc(reg->kctx->kbdev, kbase_phys_it_deref(&it), nr_pages_left,
+ page_array + reg->nr_alloc_pages);
+
+ OSK_ASSERT(pages_committed <= nr_pages_left);
+
+ if (pages_committed)
+ {
+ /* got some pages, track them */
+ kbase_mem_commit * commit;
+
+ if (reg->last_commit->allocator)
+ {
+ commit = (kbase_mem_commit*)osk_calloc(sizeof(*commit));
+ if (commit == NULL)
+ {
+ kbase_phy_pages_free(reg->kctx->kbdev, kbase_phys_it_deref(&it), pages_committed,
+ page_array + reg->nr_alloc_pages);
+ break;
+ }
+ commit->prev = reg->last_commit;
+ }
+ else
+ {
+ commit = reg->last_commit;
+ }
+
+ commit->allocator = kbase_phys_it_deref(&it);
+ commit->nr_pages = pages_committed;
+
+ reg->last_commit = commit;
+ reg->nr_alloc_pages += pages_committed;
+
+ nr_pages_left -= pages_committed;
+ }
+ }
+
+ /* no need for the iterator any more */
+ kbase_phys_it_term(&it);
+
+ if (nr_pages_left == 0)
+ {
+ return MALI_ERROR_NONE;
+ }
+ }
+
+ /* failed to allocate enough memory, roll back */
+ if (reg->nr_alloc_pages != num_pages_on_start)
+ {
+ /*we need the auxiliary var below since kbase_free_phy_pages_helper updates reg->nr_alloc_pages*/
+ u32 track_nr_alloc_pages = reg->nr_alloc_pages;
+ /* kbase_free_phy_pages_helper implicitly calls kbase_mem_usage_release_pages */
+ kbase_free_phy_pages_helper(reg, reg->nr_alloc_pages - num_pages_on_start);
+ /* Release the remaining pages */
+ kbase_mem_usage_release_pages(®->kctx->usage,
+ nr_pages_requested - (track_nr_alloc_pages - num_pages_on_start));
+ }
+ else
+ {
+ kbase_mem_usage_release_pages(®->kctx->usage, nr_pages_requested);
+ }
+ return MALI_ERROR_OUT_OF_MEMORY;
+}
+
+
+/* Frees all allocated pages of a region */
+void kbase_free_phy_pages(struct kbase_va_region *reg)
+{
+ osk_phy_addr *page_array;
+ OSK_ASSERT(NULL != reg);
+
+ page_array = kbase_get_phy_pages(reg);
+
+ if (reg->imported_type != BASE_TMEM_IMPORT_TYPE_INVALID)
+ {
+ switch (reg->imported_type)
+ {
+#if MALI_USE_UMP
+ case BASE_TMEM_IMPORT_TYPE_UMP:
+ {
+ ump_dd_handle umph;
+ umph = (ump_dd_handle)reg->imported_metadata.ump_handle;
+ ump_dd_release(umph);
+ break;
+ }
+#endif /* MALI_USE_UMP */
+#ifdef CONFIG_DMA_SHARED_BUFFER
+ case BASE_TMEM_IMPORT_TYPE_UMM:
+ {
+ dma_buf_detach(reg->imported_metadata.umm.dma_buf, reg->imported_metadata.umm.dma_attachment);
+ dma_buf_put(reg->imported_metadata.umm.dma_buf);
+ break;
+ }
+#endif /* CONFIG_DMA_SHARED_BUFFER */
+ default:
+ /* unsupported types should never reach this point */
+ OSK_ASSERT(0);
+ break;
+ }
+ reg->imported_type = BASE_TMEM_IMPORT_TYPE_INVALID;
+ }
+ else
+ {
+ if (reg->flags & KBASE_REG_IS_TB)
+ {
+ /* trace buffer being freed. Disconnect, then use osk_vfree */
+ /* save tb so we can free it after the disconnect call */
+ void * tb;
+ tb = reg->kctx->jctx.tb;
+ kbase_device_trace_buffer_uninstall(reg->kctx);
+ osk_vfree(tb);
+ }
+ else if (reg->flags & KBASE_REG_IS_RB)
+ {
+ /* nothing to do */
+ }
+ else
+ {
+ kbase_free_phy_pages_helper(reg, reg->nr_alloc_pages);
+ }
+ }
+
+ kbase_set_phy_pages(reg, NULL);
+ osk_vfree(page_array);
+}
+KBASE_EXPORT_TEST_API(kbase_free_phy_pages)
+
+int kbase_alloc_phy_pages(struct kbase_va_region *reg, u32 vsize, u32 size)
+{
+ osk_phy_addr *page_array;
+
+ OSK_ASSERT( NULL != reg );
+ OSK_ASSERT( vsize > 0 );
+
+ /* validate user provided arguments */
+ if (size > vsize || vsize > reg->nr_pages)
+ {
+ goto out_term;
+ }
+
+ /* Prevent vsize*sizeof from wrapping around.
+ * For instance, if vsize is 2**29+1, we'll allocate 1 byte and the alloc won't fail.
+ */
+ if ((size_t)vsize > ((size_t)-1 / sizeof(*page_array)))
+ {
+ goto out_term;
+ }
+
+ page_array = osk_vmalloc(vsize * sizeof(*page_array));
+ if (!page_array)
+ {
+ goto out_term;
+ }
+
+ kbase_set_phy_pages(reg, page_array);
+
+ if (MALI_ERROR_NONE != kbase_alloc_phy_pages_helper(reg, size))
+ {
+ goto out_free;
+ }
+
+ return 0;
+
+out_free:
+ osk_vfree(page_array);
+out_term:
+ return -1;
+}
+KBASE_EXPORT_TEST_API(kbase_alloc_phy_pages)
+
+/** @brief Round to +inf a tmem growable delta in pages */
+STATIC mali_bool kbasep_tmem_growable_round_delta( kbase_device *kbdev, s32 *delta_ptr )
+{
+ s32 delta;
+
+ OSK_ASSERT( delta_ptr != NULL );
+
+ delta = *delta_ptr;
+
+ if (delta >= 0)
+ {
+ u32 new_delta_unsigned = kbasep_tmem_growable_round_size( kbdev, (u32)delta );
+ if ( new_delta_unsigned > S32_MAX )
+ {
+ /* Can't support a delta of this size */
+ return MALI_FALSE;
+ }
+
+ *delta_ptr = (s32)new_delta_unsigned;
+ }
+ else
+ {
+ u32 new_delta_unsigned = (u32)-delta;
+ /* Round down */
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_9630))
+ {
+ new_delta_unsigned = new_delta_unsigned & ~(KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_HW_ISSUE_9630-1);
+ }
+ else if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8316))
+ {
+ new_delta_unsigned = new_delta_unsigned & ~(KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_HW_ISSUE_8316-1);
+ }
+ else
+ {
+ new_delta_unsigned = new_delta_unsigned & ~(KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES-1);
+ }
+
+ *delta_ptr = (s32)-new_delta_unsigned;
+ }
+
+ return MALI_TRUE;
+}
+
+mali_bool kbase_check_alloc_flags(u32 flags)
+{
+ /* At least one flags should be set */
+ if (flags == 0)
+ {
+ return MALI_FALSE;
+ }
+ /* Either the GPU or CPU must be reading from the allocated memory */
+ if ((flags & (BASE_MEM_PROT_CPU_RD | BASE_MEM_PROT_GPU_RD)) == 0)
+ {
+ return MALI_FALSE;
+ }
+ /* Either the GPU or CPU must be writing to the allocated memory */
+ if ((flags & (BASE_MEM_PROT_CPU_WR | BASE_MEM_PROT_GPU_WR)) == 0)
+ {
+ return MALI_FALSE;
+ }
+ /* GPU cannot be writing to GPU executable memory and cannot grow the memory on page fault. */
+ if ((flags & BASE_MEM_PROT_GPU_EX) && (flags & (BASE_MEM_PROT_GPU_WR | BASE_MEM_GROW_ON_GPF)))
+ {
+ return MALI_FALSE;
+ }
+ /* GPU should have at least read or write access otherwise there is no
+ reason for allocating pmem/tmem. */
+ if ((flags & (BASE_MEM_PROT_GPU_RD | BASE_MEM_PROT_GPU_WR)) == 0)
+ {
+ return MALI_FALSE;
+ }
+
+ return MALI_TRUE;
+}
+
+struct kbase_va_region *kbase_tmem_alloc(struct kbase_context *kctx,
+ u32 vsize, u32 psize,
+ u32 extent, u32 flags, mali_bool is_growable)
+{
+ struct kbase_va_region *reg;
+ mali_error err;
+ u32 align = 1;
+ u32 vsize_rounded = vsize;
+ u32 psize_rounded = psize;
+ u32 extent_rounded = extent;
+ u32 zone = KBASE_REG_ZONE_TMEM;
+
+ if ( 0 == vsize )
+ {
+ goto out1;
+ }
+
+ OSK_ASSERT(NULL != kctx);
+
+ if (!kbase_check_alloc_flags(flags))
+ {
+ goto out1;
+ }
+
+ if ((flags & BASE_MEM_GROW_ON_GPF) != MALI_FALSE)
+ {
+ /* Round up the sizes for growable on GPU page fault memory */
+ vsize_rounded = kbasep_tmem_growable_round_size( kctx->kbdev, vsize );
+ psize_rounded = kbasep_tmem_growable_round_size( kctx->kbdev, psize );
+ extent_rounded = kbasep_tmem_growable_round_size( kctx->kbdev, extent );
+
+ if ( vsize_rounded < vsize || psize_rounded < psize || extent_rounded < extent )
+ {
+ /* values too large to round */
+ return NULL;
+ }
+ }
+
+ if (flags & BASE_MEM_PROT_GPU_EX)
+ {
+ zone = KBASE_REG_ZONE_EXEC;
+ }
+
+ if ( extent > 0 && !(flags & BASE_MEM_GROW_ON_GPF))
+ {
+ OSK_PRINT_WARN(OSK_BASE_MEM, "BASE_MEM_GROW_ON_GPF flag not set when extent is greater than 0");
+ goto out1;
+ }
+
+ reg = kbase_alloc_free_region(kctx, 0, vsize_rounded, zone);
+ if (!reg)
+ {
+ goto out1;
+ }
+
+ reg->flags &= ~KBASE_REG_FREE;
+
+ kbase_update_region_flags(reg, flags, is_growable);
+
+ if (kbase_alloc_phy_pages(reg, vsize_rounded, psize_rounded))
+ {
+ goto out2;
+ }
+
+ reg->nr_alloc_pages = psize_rounded;
+ reg->extent = extent_rounded;
+
+ kbase_gpu_vm_lock(kctx);
+ err = kbase_gpu_mmap(kctx, reg, 0, vsize_rounded, align);
+ kbase_gpu_vm_unlock(kctx);
+
+ if (err)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MEM, "kbase_gpu_mmap failed\n");
+ goto out3;
+ }
+
+ return reg;
+
+out3:
+ kbase_free_phy_pages(reg);
+out2:
+ osk_free(reg);
+out1:
+ return NULL;
+}
+KBASE_EXPORT_TEST_API(kbase_tmem_alloc)
+
+mali_error kbase_tmem_resize(struct kbase_context *kctx, mali_addr64 gpu_addr, s32 delta, u32 *size, base_backing_threshold_status * failure_reason)
+{
+ kbase_va_region *reg;
+ mali_error ret = MALI_ERROR_FUNCTION_FAILED;
+#if !( (MALI_INFINITE_CACHE != 0) && !MALI_BACKEND_KERNEL )
+ /* tmem is already mapped to max_pages, so no resizing needed */
+ osk_phy_addr *phy_pages;
+#endif /* !( (MALI_INFINITE_CACHE != 0) && !MALI_BACKEND_KERNEL ) */
+ OSK_ASSERT(NULL != kctx);
+ OSK_ASSERT(size);
+ OSK_ASSERT(failure_reason);
+ OSK_ASSERT(gpu_addr != 0);
+
+ kbase_gpu_vm_lock(kctx);
+
+ /* Validate the region */
+ reg = kbase_validate_region(kctx, gpu_addr);
+ if (!reg || (reg->flags & KBASE_REG_FREE) )
+ {
+ /* not a valid region or is free memory*/
+ *failure_reason = BASE_BACKING_THRESHOLD_ERROR_INVALID_ARGUMENTS;
+ goto out_unlock;
+ }
+
+#if !( (MALI_INFINITE_CACHE != 0) && !MALI_BACKEND_KERNEL )
+ /* tmem is already mapped to max_pages, don't try to resize */
+
+ if (!( (KBASE_REG_ZONE_MASK & reg->flags) == KBASE_REG_ZONE_TMEM ||
+ (KBASE_REG_ZONE_MASK & reg->flags) == KBASE_REG_ZONE_EXEC ) )
+ {
+ /* not a valid region - not tmem or exec region */
+ *failure_reason = BASE_BACKING_THRESHOLD_ERROR_INVALID_ARGUMENTS;
+ goto out_unlock;
+ }
+ if (0 == (reg->flags & KBASE_REG_GROWABLE))
+ {
+ /* not growable */
+ *failure_reason = BASE_BACKING_THRESHOLD_ERROR_NOT_GROWABLE;
+ goto out_unlock;
+ }
+
+ if ( (delta != 0) && !OSK_DLIST_IS_EMPTY(®->map_list))
+ {
+ /* We still have mappings */
+ *failure_reason = BASE_BACKING_THRESHOLD_ERROR_MAPPED;
+ goto out_unlock;
+ }
+
+ if ( reg->flags & KBASE_REG_PF_GROW )
+ {
+ /* Apply rounding to +inf on the delta, which may cause a negative delta to become zero */
+ if ( kbasep_tmem_growable_round_delta( kctx->kbdev, &delta ) == MALI_FALSE )
+ {
+ /* Can't round this big a delta */
+ *failure_reason = BASE_BACKING_THRESHOLD_ERROR_INVALID_ARGUMENTS;
+ goto out_unlock;
+ }
+ }
+
+ if (delta < 0 && (u32)-delta > reg->nr_alloc_pages)
+ {
+ /* Underflow */
+ *failure_reason = BASE_BACKING_THRESHOLD_ERROR_INVALID_ARGUMENTS;
+ goto out_unlock;
+ }
+ if (reg->nr_alloc_pages + delta > reg->nr_pages)
+ {
+ /* Would overflow the VA region */
+ *failure_reason = BASE_BACKING_THRESHOLD_ERROR_INVALID_ARGUMENTS;
+ goto out_unlock;
+ }
+
+ phy_pages = kbase_get_phy_pages(reg);
+
+ if (delta > 0)
+ {
+ mali_error err;
+
+ /* Allocate some more pages */
+ if (MALI_ERROR_NONE != kbase_alloc_phy_pages_helper(reg, delta))
+ {
+ *failure_reason = BASE_BACKING_THRESHOLD_ERROR_OOM;
+ goto out_unlock;
+ }
+ err = kbase_mmu_insert_pages(kctx, reg->start_pfn + reg->nr_alloc_pages - delta,
+ phy_pages + reg->nr_alloc_pages - delta,
+ delta, reg->flags);
+ if(MALI_ERROR_NONE != err)
+ {
+ kbase_free_phy_pages_helper(reg, delta);
+ *failure_reason = BASE_BACKING_THRESHOLD_ERROR_OOM;
+ goto out_unlock;
+ }
+ }
+ else if (delta < 0)
+ {
+ mali_error err;
+ /* Free some pages */
+
+ /* Get the absolute value of delta. Note that we have to add one before and after the negation to avoid
+ * overflowing when delta is INT_MIN */
+ u32 num_pages = (u32)(-(delta+1))+1;
+
+ err = kbase_mmu_teardown_pages(kctx, reg->start_pfn + reg->nr_alloc_pages - num_pages, num_pages);
+ if(MALI_ERROR_NONE != err)
+ {
+ *failure_reason = BASE_BACKING_THRESHOLD_ERROR_OOM;
+ goto out_unlock;
+ }
+
+ if (kbase_hw_has_issue(kctx->kbdev, BASE_HW_ISSUE_6367))
+ {
+ /* Wait for GPU to flush write buffer before freeing physical pages */
+ kbase_wait_write_flush(kctx);
+ }
+
+ kbase_free_phy_pages_helper(reg, num_pages);
+ }
+ /* else just a size query */
+
+#endif /* !( (MALI_INFINITE_CACHE != 0) && !MALI_BACKEND_KERNEL ) */
+
+ *size = reg->nr_alloc_pages;
+
+ ret = MALI_ERROR_NONE;
+
+out_unlock:
+ kbase_gpu_vm_unlock(kctx);
+ return ret;
+}
+KBASE_EXPORT_TEST_API(kbase_tmem_resize)
+
+#if MALI_USE_UMP
+
+static struct kbase_va_region *kbase_tmem_from_ump(struct kbase_context *kctx, ump_secure_id id, u64 * const pages)
+{
+ struct kbase_va_region *reg;
+ mali_error err;
+ ump_dd_handle umph;
+ u64 vsize;
+ u64 block_count;
+ const ump_dd_physical_block_64 * block_array;
+ osk_phy_addr *page_array;
+ u64 i, j;
+ int page = 0;
+ ump_alloc_flags ump_flags;
+ ump_alloc_flags cpu_flags;
+ ump_alloc_flags gpu_flags;
+
+ OSK_ASSERT(NULL != pages);
+
+ umph = ump_dd_from_secure_id(id);
+ if (UMP_DD_INVALID_MEMORY_HANDLE == umph)
+ {
+ return NULL;
+ }
+
+ ump_flags = ump_dd_allocation_flags_get(umph);
+ cpu_flags = (ump_flags >> UMP_DEVICE_CPU_SHIFT) & UMP_DEVICE_MASK;
+ gpu_flags = (ump_flags >> kctx->kbdev->memdev.ump_device_id) & UMP_DEVICE_MASK;
+
+ vsize = ump_dd_size_get_64(umph);
+ vsize >>= OSK_PAGE_SHIFT;
+
+ reg = kbase_alloc_free_region(kctx, 0, vsize, KBASE_REG_ZONE_TMEM);
+ if (!reg)
+ {
+ goto out1;
+ }
+
+ reg->flags &= ~KBASE_REG_FREE;
+ reg->flags |= KBASE_REG_GPU_NX; /* UMP is always No eXecute */
+ reg->flags &= ~KBASE_REG_GROWABLE; /* UMP cannot be grown */
+
+ reg->imported_type = BASE_TMEM_IMPORT_TYPE_UMP;
+
+ reg->imported_metadata.ump_handle = umph;
+
+ if ((cpu_flags & (UMP_HINT_DEVICE_RD|UMP_HINT_DEVICE_WR)) == (UMP_HINT_DEVICE_RD|UMP_HINT_DEVICE_WR))
+ {
+ reg->flags |= KBASE_REG_CPU_CACHED;
+ }
+
+ if (cpu_flags & UMP_PROT_DEVICE_WR)
+ {
+ reg->flags |= KBASE_REG_CPU_WR;
+ }
+
+ if (cpu_flags & UMP_PROT_DEVICE_RD)
+ {
+ reg->flags |= KBASE_REG_CPU_RD;
+ }
+
+
+ if ((gpu_flags & (UMP_HINT_DEVICE_RD|UMP_HINT_DEVICE_WR)) == (UMP_HINT_DEVICE_RD|UMP_HINT_DEVICE_WR))
+ {
+ reg->flags |= KBASE_REG_GPU_CACHED;
+ }
+
+ if (gpu_flags & UMP_PROT_DEVICE_WR)
+ {
+ reg->flags |= KBASE_REG_GPU_WR;
+ }
+
+ if (gpu_flags & UMP_PROT_DEVICE_RD)
+ {
+ reg->flags |= KBASE_REG_GPU_RD;
+ }
+
+ /* ump phys block query */
+ ump_dd_phys_blocks_get_64(umph, &block_count, &block_array);
+
+ page_array = osk_vmalloc(vsize * sizeof(*page_array));
+ if (!page_array)
+ {
+ goto out2;
+ }
+
+ for (i = 0; i < block_count; i++)
+ {
+ for (j = 0; j < (block_array[i].size >> OSK_PAGE_SHIFT); j++)
+ {
+ page_array[page] = block_array[i].addr + (j << OSK_PAGE_SHIFT);
+ page++;
+ }
+ }
+
+ kbase_set_phy_pages(reg, page_array);
+
+ reg->nr_alloc_pages = vsize;
+ reg->extent = vsize;
+
+ kbase_gpu_vm_lock(kctx);
+ err = kbase_gpu_mmap(kctx, reg, 0, vsize, 1/* no alignment */);
+ kbase_gpu_vm_unlock(kctx);
+
+ if (err)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MEM, "kbase_gpu_mmap failed\n");
+ goto out3;
+ }
+
+ *pages = vsize;
+
+ return reg;
+
+out3:
+ osk_vfree(page_array);
+out2:
+ osk_free(reg);
+out1:
+ ump_dd_release(umph);
+ return NULL;
+
+
+}
+
+#endif /* MALI_USE_UMP */
+
+#ifdef CONFIG_DMA_SHARED_BUFFER
+static struct kbase_va_region *kbase_tmem_from_umm(struct kbase_context *kctx, int fd, u64 * const pages)
+{
+ struct kbase_va_region *reg;
+ struct dma_buf * dma_buf;
+ struct dma_buf_attachment * dma_attachment;
+ osk_phy_addr *page_array;
+ unsigned long nr_pages;
+ mali_error err;
+
+ dma_buf = dma_buf_get(fd);
+ if (IS_ERR_OR_NULL(dma_buf))
+ goto no_buf;
+
+ dma_attachment = dma_buf_attach(dma_buf, kctx->kbdev->osdev.dev);
+ if (!dma_attachment)
+ goto no_attachment;
+
+ nr_pages = (dma_buf->size + PAGE_SIZE - 1) >> PAGE_SHIFT;
+
+ reg = kbase_alloc_free_region(kctx, 0, nr_pages , KBASE_REG_ZONE_TMEM);
+ if (!reg)
+ goto no_region;
+
+ reg->flags &= ~KBASE_REG_FREE;
+ reg->flags |= KBASE_REG_GPU_NX; /* UMM is always No eXecute */
+ reg->flags &= ~KBASE_REG_GROWABLE; /* UMM cannot be grown */
+ reg->flags |= KBASE_REG_NO_CPU_MAP; /* UMM can't be mapped on the CPU *//*TODO: GP this flag to be removed to allow mapping to CPU*/
+
+ reg->flags |= KBASE_REG_GPU_CACHED;
+
+ /* no read or write permission given on import, only on run do we give the right permissions */
+
+ reg->imported_type = BASE_TMEM_IMPORT_TYPE_UMM;
+
+ reg->imported_metadata.umm.st = NULL;
+ reg->imported_metadata.umm.dma_buf = dma_buf;
+ reg->imported_metadata.umm.dma_attachment = dma_attachment;
+ reg->imported_metadata.umm.current_mapping_usage_count = 0;
+
+ page_array = osk_vmalloc(nr_pages * sizeof(*page_array));
+ if (!page_array)
+ goto no_page_array;
+
+ OSK_MEMSET(page_array, 0, nr_pages * sizeof(*page_array));
+
+ kbase_set_phy_pages(reg, page_array);
+
+ reg->nr_alloc_pages = nr_pages;
+ reg->extent = nr_pages;
+
+ kbase_gpu_vm_lock(kctx);
+ err = kbase_add_va_region(kctx, reg, 0, nr_pages, 1);
+ kbase_gpu_vm_unlock(kctx);
+ if (err != MALI_ERROR_NONE)
+ goto no_addr_reserve;
+
+ *pages = nr_pages;
+
+ return reg;
+
+no_addr_reserve:
+ osk_vfree(page_array);
+no_page_array:
+ osk_free(reg);
+no_region:
+ dma_buf_detach(dma_buf, dma_attachment);
+no_attachment:
+ dma_buf_put(dma_buf);
+no_buf:
+ return NULL;
+}
+#endif /* CONFIG_DMA_SHARED_BUFFER */
+
+struct kbase_va_region *kbase_tmem_import(struct kbase_context *kctx, base_tmem_import_type type, int handle, u64 * const pages)
+{
+ switch (type)
+ {
+#if MALI_USE_UMP
+ case BASE_TMEM_IMPORT_TYPE_UMP:
+ {
+ ump_secure_id id;
+ id = (ump_secure_id)handle;
+ return kbase_tmem_from_ump(kctx, id, pages);
+ }
+#endif /* MALI_USE_UMP */
+#ifdef CONFIG_DMA_SHARED_BUFFER
+ case BASE_TMEM_IMPORT_TYPE_UMM:
+ return kbase_tmem_from_umm(kctx, handle, pages);
+#endif /* CONFIG_DMA_SHARED_BUFFER */
+ default:
+ return NULL;
+ }
+}
+
+/**
+ * @brief Acquire the per-context region list lock
+ */
+void kbase_gpu_vm_lock(struct kbase_context *kctx)
+{
+ OSK_ASSERT(kctx != NULL);
+ osk_mutex_lock(&kctx->reg_lock);
+}
+KBASE_EXPORT_TEST_API(kbase_gpu_vm_lock)
+
+/**
+ * @brief Release the per-context region list lock
+ */
+void kbase_gpu_vm_unlock(struct kbase_context *kctx)
+{
+ OSK_ASSERT(kctx != NULL);
+ osk_mutex_unlock(&kctx->reg_lock);
+}
+KBASE_EXPORT_TEST_API(kbase_gpu_vm_unlock)
+
+/* will be called during init time only */
+mali_error kbase_register_memory_regions(kbase_device * kbdev, const kbase_attribute *attributes)
+{
+ int total_regions;
+ int dedicated_regions;
+ int allocators_initialized;
+ osk_phy_allocator * allocs;
+ kbase_memory_performance shared_memory_performance;
+ kbasep_memory_region_performance *region_performance;
+ kbase_memory_resource *resource;
+ const kbase_attribute *current_attribute;
+ u32 max_shared_memory;
+ kbasep_mem_device * memdev;
+
+ OSK_ASSERT(kbdev);
+ OSK_ASSERT(attributes);
+
+ memdev = &kbdev->memdev;
+
+ /* Programming error to register memory after we've started using the iterator interface */
+#if MALI_DEBUG
+ OSK_ASSERT(memdev->allocators.it_bound == MALI_FALSE);
+#endif /* MALI_DEBUG */
+
+ max_shared_memory = (u32) kbasep_get_config_value(kbdev, attributes, KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_MAX);
+ shared_memory_performance =
+ (kbase_memory_performance)kbasep_get_config_value(kbdev, attributes, KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_PERF_GPU);
+ /* count dedicated_memory_regions */
+ dedicated_regions = kbasep_get_config_attribute_count_by_id(attributes, KBASE_CONFIG_ATTR_MEMORY_RESOURCE);
+
+ total_regions = dedicated_regions;
+ if (max_shared_memory > 0)
+ {
+ total_regions++;
+ }
+
+ if (total_regions == 0)
+ {
+ OSK_PRINT_ERROR(OSK_BASE_MEM, "No memory regions specified");
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ region_performance = osk_malloc(sizeof(kbasep_memory_region_performance) * total_regions);
+
+ if (region_performance == NULL)
+ {
+ goto out;
+ }
+
+ allocs = osk_malloc(sizeof(osk_phy_allocator) * total_regions);
+ if (allocs == NULL)
+ {
+ goto out_perf;
+ }
+
+ current_attribute = attributes;
+ allocators_initialized = 0;
+ while (current_attribute != NULL)
+ {
+ current_attribute = kbasep_get_next_attribute(current_attribute, KBASE_CONFIG_ATTR_MEMORY_RESOURCE);
+
+ if (current_attribute != NULL)
+ {
+ resource = (kbase_memory_resource *)current_attribute->data;
+ if (OSK_ERR_NONE != osk_phy_allocator_init(&allocs[allocators_initialized], resource->base,
+ (u32)(resource->size >> OSK_PAGE_SHIFT), resource->name))
+ {
+ goto out_allocator_term;
+ }
+
+ kbasep_get_memory_performance(resource, ®ion_performance[allocators_initialized].cpu_performance,
+ ®ion_performance[allocators_initialized].gpu_performance);
+ current_attribute++;
+ allocators_initialized++;
+ }
+ }
+
+ /* register shared memory region */
+ if (max_shared_memory > 0)
+ {
+ if (OSK_ERR_NONE != osk_phy_allocator_init(&allocs[allocators_initialized], 0,
+ max_shared_memory >> OSK_PAGE_SHIFT, NULL))
+ {
+ goto out_allocator_term;
+ }
+
+ region_performance[allocators_initialized].cpu_performance = KBASE_MEM_PERF_NORMAL;
+ region_performance[allocators_initialized].gpu_performance = shared_memory_performance;
+ allocators_initialized++;
+ }
+
+ if (MALI_ERROR_NONE != kbase_mem_usage_init(&memdev->usage, max_shared_memory >> OSK_PAGE_SHIFT))
+ {
+ goto out_allocator_term;
+ }
+
+ if (MALI_ERROR_NONE != kbasep_allocator_order_list_create(allocs, region_performance, total_regions, memdev->allocators.sorted_allocs,
+ ALLOCATOR_ORDER_COUNT))
+ {
+ goto out_memctx_term;
+ }
+
+ memdev->allocators.allocs = allocs;
+ memdev->allocators.count = total_regions;
+
+ osk_free(region_performance);
+
+ return MALI_ERROR_NONE;
+
+out_memctx_term:
+ kbase_mem_usage_term(&memdev->usage);
+out_allocator_term:
+ while (allocators_initialized-- > 0)
+ {
+ osk_phy_allocator_term(&allocs[allocators_initialized]);
+ }
+ osk_free(allocs);
+out_perf:
+ osk_free(region_performance);
+out:
+ return MALI_ERROR_OUT_OF_MEMORY;
+}
+KBASE_EXPORT_TEST_API(kbase_register_memory_regions)
+
+static mali_error kbasep_allocator_order_list_create( osk_phy_allocator * allocators,
+ kbasep_memory_region_performance *region_performance, int memory_region_count,
+ osk_phy_allocator ***sorted_allocs, int allocator_order_count)
+{
+ int performance;
+ int regions_sorted;
+ int i;
+ void *sorted_alloc_mem_block;
+
+ sorted_alloc_mem_block = osk_malloc(sizeof(osk_phy_allocator **) * memory_region_count * allocator_order_count);
+ if (sorted_alloc_mem_block == NULL)
+ {
+ goto out;
+ }
+
+ /* each allocator list points to memory in recently allocated block */
+ for (i = 0; i < ALLOCATOR_ORDER_COUNT; i++)
+ {
+ sorted_allocs[i] = (osk_phy_allocator **)sorted_alloc_mem_block + memory_region_count*i;
+ }
+
+ /* use the same order as in config file */
+ for (i = 0; i < memory_region_count; i++)
+ {
+ sorted_allocs[ALLOCATOR_ORDER_CONFIG][i] = &allocators[i];
+ }
+
+ /* Sort allocators by GPU performance */
+ performance = KBASE_MEM_PERF_FAST;
+ regions_sorted = 0;
+ while (performance >= KBASE_MEM_PERF_SLOW)
+ {
+ for (i = 0; i < memory_region_count; i++)
+ {
+ if (region_performance[i].gpu_performance == (kbase_memory_performance)performance)
+ {
+ sorted_allocs[ALLOCATOR_ORDER_GPU_PERFORMANCE][regions_sorted] = &allocators[i];
+ regions_sorted++;
+ }
+ }
+ performance--;
+ }
+
+ /* Sort allocators by CPU performance */
+ performance = KBASE_MEM_PERF_FAST;
+ regions_sorted = 0;
+ while (performance >= KBASE_MEM_PERF_SLOW)
+ {
+ for (i = 0; i < memory_region_count; i++)
+ {
+ if ((int)region_performance[i].cpu_performance == performance)
+ {
+ sorted_allocs[ALLOCATOR_ORDER_CPU_PERFORMANCE][regions_sorted] = &allocators[i];
+ regions_sorted++;
+ }
+ }
+ performance--;
+ }
+
+ /* Sort allocators by CPU and GPU performance (equally important) */
+ performance = 2 * KBASE_MEM_PERF_FAST;
+ regions_sorted = 0;
+ while (performance >= 2*KBASE_MEM_PERF_SLOW)
+ {
+ for (i = 0; i < memory_region_count; i++)
+ {
+ if ((int)(region_performance[i].cpu_performance + region_performance[i].gpu_performance) == performance)
+ {
+ sorted_allocs[ALLOCATOR_ORDER_CPU_GPU_PERFORMANCE][regions_sorted] = &allocators[i];
+ regions_sorted++;
+ }
+ }
+ performance--;
+ }
+ return MALI_ERROR_NONE;
+out:
+ return MALI_ERROR_OUT_OF_MEMORY;
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_mem.h
+ * Base kernel memory APIs
+ */
+
+#ifndef _KBASE_MEM_H_
+#define _KBASE_MEM_H_
+
+#ifndef _KBASE_H_
+#error "Don't include this file directly, use mali_kbase.h instead"
+#endif
+
+#include <malisw/mali_malisw.h>
+#include <osk/mali_osk.h>
+#if MALI_USE_UMP
+#ifndef __KERNEL__
+#include <ump/src/library/common/ump_user.h>
+#endif
+#include <ump/ump_kernel_interface.h>
+#endif /* MALI_USE_UMP */
+#include <kbase/mali_base_kernel.h>
+#include <kbase/src/common/mali_kbase_hw.h>
+#include "mali_kbase_pm.h"
+#include "mali_kbase_defs.h"
+
+/* Part of the workaround for uTLB invalid pages is to ensure we grow/shrink tmem by 4 pages at a time */
+#define KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_LOG2_HW_ISSUE_8316 (2) /* round to 4 pages */
+
+/* Part of the workaround for PRLAM-9630 requires us to grow/shrink memory by 8 pages.
+The MMU reads in 8 page table entries from memory at a time, if we have more than one page fault within the same 8 pages and
+page tables are updated accordingly, the MMU does not re-read the page table entries from memory for the subsequent page table
+updates and generates duplicate page faults as the page table information used by the MMU is not valid. */
+#define KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_LOG2_HW_ISSUE_9630 (3) /* round to 8 pages */
+
+#define KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_LOG2 (0) /* round to 1 page */
+
+/* This must always be a power of 2 */
+#define KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES (1u << KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_LOG2)
+#define KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_HW_ISSUE_8316 (1u << KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_LOG2_HW_ISSUE_8316)
+#define KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_HW_ISSUE_9630 (1u << KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_LOG2_HW_ISSUE_9630)
+/**
+ * A CPU mapping
+ */
+typedef struct kbase_cpu_mapping
+{
+ osk_dlist_item link;
+ osk_virt_addr uaddr;
+ u32 nr_pages;
+ mali_size64 page_off;
+ void *private; /* Use for VMA */
+} kbase_cpu_mapping;
+
+/**
+ * A physical memory (sub-)commit
+ */
+typedef struct kbase_mem_commit
+{
+ osk_phy_allocator * allocator;
+ u32 nr_pages;
+ struct kbase_mem_commit * prev;
+ /*
+ * The offset of the commit is implict by
+ * the prev_commit link position of this node
+ */
+} kbase_mem_commit;
+
+/**
+ * A GPU memory region, and attributes for CPU mappings.
+ */
+typedef struct kbase_va_region
+{
+ osk_dlist_item link;
+
+ struct kbase_context *kctx; /* Backlink to base context */
+
+ u64 start_pfn; /* The PFN in GPU space */
+ u32 nr_pages; /* VA size */
+
+#define KBASE_REG_FREE (1ul << 0) /* Free region */
+#define KBASE_REG_CPU_WR (1ul << 1) /* CPU write access */
+#define KBASE_REG_GPU_WR (1ul << 2) /* GPU write access */
+#define KBASE_REG_GPU_NX (1ul << 3) /* No eXectue flag */
+#define KBASE_REG_CPU_CACHED (1ul << 4) /* Is CPU cached? */
+#define KBASE_REG_GPU_CACHED (1ul << 5) /* Is GPU cached? */
+
+#define KBASE_REG_GROWABLE (1ul << 6) /* Is growable? */
+#define KBASE_REG_PF_GROW (1ul << 7) /* Can grow on pf? */
+
+#define KBASE_REG_IS_RB (1ul << 8) /* Is ringbuffer? */
+#define KBASE_REG_IS_MMU_DUMP (1ul << 9) /* Is an MMU dump */
+#define KBASE_REG_IS_TB (1ul << 10) /* Is register trace buffer? */
+
+#define KBASE_REG_SHARE_IN (1ul << 11) /* inner shareable coherency */
+#define KBASE_REG_SHARE_BOTH (1ul << 12) /* inner & outer shareable coherency */
+
+#define KBASE_REG_NO_CPU_MAP (1ul << 13) /* buffer can not be mapped on the CPU, only available for the GPU */
+
+#define KBASE_REG_ZONE_MASK (3ul << 14) /* Space for 4 different zones */
+#define KBASE_REG_ZONE(x) (((x) & 3) << 14)
+
+#define KBASE_REG_GPU_RD (1ul<<16) /* GPU write access */
+#define KBASE_REG_CPU_RD (1ul<<17) /* CPU read access */
+
+#define KBASE_REG_FLAGS_NR_BITS 18 /* Number of bits used by kbase_va_region flags */
+
+#define KBASE_REG_ZONE_PMEM KBASE_REG_ZONE(0)
+
+#ifndef KBASE_REG_ZONE_TMEM /* To become 0 on a 64bit platform */
+/*
+ * On a 32bit platform, TMEM should be wired from 4GB to the VA limit
+ * of the GPU, which is currently hardcoded at 48 bits. Unfortunately,
+ * the Linux mmap() interface limits us to 2^32 pages (2^44 bytes, see
+ * mmap64 man page for reference).
+ */
+#define KBASE_REG_ZONE_EXEC KBASE_REG_ZONE(1) /* Dedicated 4GB region for shader code */
+#define KBASE_REG_ZONE_EXEC_BASE ((1ULL << 32) >> OSK_PAGE_SHIFT)
+#define KBASE_REG_ZONE_EXEC_SIZE (((1ULL << 33) >> OSK_PAGE_SHIFT) - \
+ KBASE_REG_ZONE_EXEC_BASE)
+
+#define KBASE_REG_ZONE_TMEM KBASE_REG_ZONE(2)
+#define KBASE_REG_ZONE_TMEM_BASE ((1ULL << 33) >> OSK_PAGE_SHIFT) /* Starting after KBASE_REG_ZONE_EXEC */
+#define KBASE_REG_ZONE_TMEM_SIZE (((1ULL << 44) >> OSK_PAGE_SHIFT) - \
+ KBASE_REG_ZONE_TMEM_BASE)
+#endif
+
+#define KBASE_REG_COOKIE_MASK (~((1ul << KBASE_REG_FLAGS_NR_BITS)-1))
+#define KBASE_REG_COOKIE(x) ((x << KBASE_REG_FLAGS_NR_BITS) & KBASE_REG_COOKIE_MASK)
+
+/* Bit mask of cookies that not used for PMEM but reserved for other uses */
+#define KBASE_REG_RESERVED_COOKIES 7ULL
+/* The reserved cookie values */
+#define KBASE_REG_COOKIE_RB 0
+#define KBASE_REG_COOKIE_MMU_DUMP 1
+#define KBASE_REG_COOKIE_TB 2
+
+ u32 flags;
+
+ u32 nr_alloc_pages; /* nr of pages allocated */
+ u32 extent; /* nr of pages alloc'd on PF */
+
+ /* two variables to track our physical commits: */
+
+ /* We always have a root commit.
+ * Most allocation will only have this one.
+ * */
+ kbase_mem_commit root_commit;
+
+ /* This one is initialized to point to the root_commit,
+ * but if a new and separate commit is needed it will point
+ * to the last (still valid) commit we've done */
+ kbase_mem_commit * last_commit;
+
+ osk_phy_addr *phy_pages;
+
+ osk_dlist map_list;
+
+ /* non-NULL if this memory object is a kds_resource */
+ struct kds_resource * kds_res;
+
+ base_tmem_import_type imported_type;
+
+ /* member in union valid based on imported_type */
+ union
+ {
+#if MALI_USE_UMP == 1
+ ump_dd_handle ump_handle;
+#endif /*MALI_USE_UMP == 1*/
+#ifdef CONFIG_DMA_SHARED_BUFFER
+ struct
+ {
+ struct dma_buf * dma_buf;
+ struct dma_buf_attachment * dma_attachment;
+ unsigned int current_mapping_usage_count;
+ struct sg_table * st;
+ } umm;
+#endif /* CONFIG_DMA_SHARED_BUFFER */
+ } imported_metadata;
+
+} kbase_va_region;
+
+/* Common functions */
+static INLINE osk_phy_addr *kbase_get_phy_pages(struct kbase_va_region *reg)
+{
+ OSK_ASSERT(reg);
+
+ return reg->phy_pages;
+}
+
+static INLINE void kbase_set_phy_pages(struct kbase_va_region *reg, osk_phy_addr *phy_pages)
+{
+ OSK_ASSERT(reg);
+
+ reg->phy_pages = phy_pages;
+}
+
+/**
+ * @brief Allocate physical memory and track shared OS memory usage.
+ *
+ * This function is kbase wrapper of osk_phy_pages_alloc. Apart from allocating memory it also tracks shared OS memory
+ * usage and fails whenever shared memory limits would be exceeded.
+ *
+ * @param[in] kbdev pointer to kbase_device structure for which memory is allocated
+ * @param[in] allocator initialized physical allocator
+ * @param[in] nr_pages number of physical pages to allocate
+ * @param[out] pages array of \a nr_pages elements storing the physical
+ * address of an allocated page
+ * @return The number of pages successfully allocated,
+ * which might be lower than requested, including zero pages.
+ *
+ * @see ::osk_phy_pages_alloc
+ */
+u32 kbase_phy_pages_alloc(struct kbase_device *kbdev, osk_phy_allocator *allocator, u32 nr_pages, osk_phy_addr *pages);
+
+/**
+ * @brief Free physical memory and track shared memory usage
+ *
+ * This function, like osk_phy_pages_free, frees physical memory but also tracks shared OS memory usage.
+ *
+ * @param[in] kbdev pointer to kbase_device for which memory is allocated
+ * @param[in] allocator initialized physical allocator
+ * @param[in] nr_pages number of physical pages to allocate
+ * @param[out] pages array of \a nr_pages elements storing the physical
+ * address of an allocated page
+ *
+ * @see ::osk_phy_pages_free
+ */
+void kbase_phy_pages_free(struct kbase_device *kbdev, osk_phy_allocator *allocator, u32 nr_pages, osk_phy_addr *pages);
+
+/**
+ * @brief Register shared and dedicated memory regions
+ *
+ * Function registers shared and dedicated memory regions (registers physical allocator for each region)
+ * using given configuration attributes. Additionally, several ordered lists of physical allocators are created with
+ * different sort order (based on CPU, GPU, CPU+GPU performance and order in config). If there are many memory regions
+ * with the same performance, then order in which they appeared in config is important. Shared OS memory is treated as if
+ * it's defined after dedicated memory regions, so unless it matches region's performance flags better, it's chosen last.
+ *
+ * @param[in] kbdev pointer to kbase_device for which regions are registered
+ * @param[in] attributes array of configuration attributes. It must be terminated with KBASE_CONFIG_ATTR_END attribute
+ *
+ * @return MALI_ERROR_NONE if no error occurred. Error code otherwise
+ *
+ * @see ::kbase_alloc_phy_pages_helper
+ */
+mali_error kbase_register_memory_regions(kbase_device * kbdev, const kbase_attribute *attributes);
+
+/**
+ * @brief Frees memory regions registered for the given device.
+ *
+ * @param[in] kbdev pointer to kbase device for which memory regions are to be freed
+ */
+void kbase_free_memory_regions(kbase_device * kbdev);
+
+mali_error kbase_mem_init(kbase_device * kbdev);
+void kbase_mem_halt(kbase_device * kbdev);
+void kbase_mem_term(kbase_device * kbdev);
+
+
+/**
+ * @brief Initializes memory context which tracks memory usage.
+ *
+ * Function initializes memory context with given max_pages value.
+ *
+ * @param[in] usage usage tracker
+ * @param[in] max_pages maximum pages allowed to be allocated within this memory context
+ *
+ * @return MALI_ERROR_NONE in case of error. Error code otherwise.
+ */
+mali_error kbase_mem_usage_init(kbasep_mem_usage * usage, u32 max_pages);
+
+/*
+ * @brief Terminates given memory context
+ *
+ * @param[in] usage usage tracker
+ *
+ * @return MALI_ERROR_NONE in case of error. Error code otherwise.
+ */
+void kbase_mem_usage_term(kbasep_mem_usage *usage);
+
+/*
+ * @brief Requests a number of pages from the given context.
+ *
+ * Function requests a number of pages from the given context. Context is updated only if it contains enough number of
+ * free pages. Otherwise function returns error and no pages are claimed.
+ *
+ * @param[in] usage usage tracker
+ * @param[in] nr_pages number of pages requested
+ *
+ * @return MALI_ERROR_NONE when context page request succeeded. Error code otherwise.
+ */
+mali_error kbase_mem_usage_request_pages(kbasep_mem_usage *usage, u32 nr_pages);
+
+/*
+ * @brief Release a number of pages from the given context.
+ *
+ * @param[in] usage usage tracker
+ * @param[in] nr_pages number of pages to be released
+ */
+void kbase_mem_usage_release_pages(kbasep_mem_usage *usage, u32 nr_pages);
+
+struct kbase_va_region *kbase_alloc_free_region(struct kbase_context *kctx, u64 start_pfn, u32 nr_pages, u32 zone);
+void kbase_free_alloced_region(struct kbase_va_region *reg);
+mali_error kbase_add_va_region(struct kbase_context *kctx,
+ struct kbase_va_region *reg,
+ mali_addr64 addr, u32 nr_pages,
+ u32 align);
+kbase_va_region *kbase_region_lookup(kbase_context *kctx, mali_addr64 gpu_addr);
+
+mali_error kbase_gpu_mmap(struct kbase_context *kctx,
+ struct kbase_va_region *reg,
+ mali_addr64 addr, u32 nr_pages,
+ u32 align);
+mali_bool kbase_check_alloc_flags(u32 flags);
+void kbase_update_region_flags(struct kbase_va_region *reg, u32 flags, mali_bool is_growable);
+
+void kbase_gpu_vm_lock(struct kbase_context *kctx);
+void kbase_gpu_vm_unlock(struct kbase_context *kctx);
+
+void kbase_free_phy_pages(struct kbase_va_region *reg);
+int kbase_alloc_phy_pages(struct kbase_va_region *reg, u32 vsize, u32 size);
+
+mali_error kbase_cpu_free_mapping(struct kbase_va_region *reg, const void *ptr);
+
+mali_error kbase_mmu_init(struct kbase_context *kctx);
+void kbase_mmu_term(struct kbase_context *kctx);
+
+osk_phy_addr kbase_mmu_alloc_pgd(kbase_context *kctx);
+void kbase_mmu_free_pgd(struct kbase_context *kctx);
+mali_error kbase_mmu_insert_pages(struct kbase_context *kctx, u64 vpfn,
+ osk_phy_addr *phys, u32 nr, u32 flags);
+mali_error kbase_mmu_teardown_pages(struct kbase_context *kctx, u64 vpfn, u32 nr);
+
+/**
+ * @brief Check that a pointer is actually a valid region.
+ *
+ * Must be called with context lock held.
+ */
+struct kbase_va_region *kbase_validate_region(struct kbase_context *kctx, mali_addr64 gpu_addr);
+
+/**
+ * @brief Register region and map it on the GPU.
+ *
+ * Call kbase_add_va_region() and map the region on the GPU.
+ */
+mali_error kbase_gpu_mmap(struct kbase_context *kctx,
+ struct kbase_va_region *reg,
+ mali_addr64 addr, u32 nr_pages,
+ u32 align);
+
+/**
+ * @brief Remove the region from the GPU and unregister it.
+ *
+ * Must be called with context lock held.
+ */
+mali_error kbase_gpu_munmap(struct kbase_context *kctx, struct kbase_va_region *reg);
+
+/**
+ * The caller has the following locking conditions:
+ * - It must hold kbase_as::transaction_mutex on kctx's address space
+ * - It must hold the kbasep_js_device_data::runpool_irq::lock
+ */
+void kbase_mmu_update(struct kbase_context *kctx);
+
+/**
+ * The caller has the following locking conditions:
+ * - It must hold kbase_as::transaction_mutex on kctx's address space
+ * - It must hold the kbasep_js_device_data::runpool_irq::lock
+ */
+void kbase_mmu_disable (kbase_context *kctx);
+
+void kbase_mmu_interrupt(kbase_device * kbdev, u32 irq_stat);
+
+/**
+ * @brief Allocates physical pages using registered physical allocators.
+ *
+ * Function allocates physical pages using registered physical allocators. Allocator list is iterated until all pages
+ * are successfully allocated. Function tries to match the most appropriate order of iteration basing on
+ * KBASE_REG_CPU_CACHED and KBASE_REG_GPU_CACHED flags of the region.
+ *
+ * @param[in] reg memory region in which physical pages are supposed to be allocated
+ * @param[in] nr_pages number of physical pages to allocate
+ *
+ * @return MALI_ERROR_NONE if all pages have been successfully allocated. Error code otherwise
+ *
+ * @see kbase_register_memory_regions
+ */
+mali_error kbase_alloc_phy_pages_helper(kbase_va_region *reg, u32 nr_pages);
+
+/** Dump the MMU tables to a buffer
+ *
+ * This function allocates a buffer (of @c nr_pages pages) to hold a dump of the MMU tables and fills it. If the
+ * buffer is too small then the return value will be NULL.
+ *
+ * The GPU vm lock must be held when calling this function.
+ *
+ * The buffer returned should be freed with @ref osk_vfree when it is no longer required.
+ *
+ * @param[in] kctx The kbase context to dump
+ * @param[in] nr_pages The number of pages to allocate for the buffer.
+ *
+ * @return The address of the buffer containing the MMU dump or NULL on error (including if the @c nr_pages is too
+ * small)
+ */
+void *kbase_mmu_dump(struct kbase_context *kctx,int nr_pages);
+
+mali_error kbase_sync_now(kbase_context *kctx, base_syncset *syncset);
+void kbase_pre_job_sync(kbase_context *kctx, base_syncset *syncsets, u32 nr);
+void kbase_post_job_sync(kbase_context *kctx, base_syncset *syncsets, u32 nr);
+
+struct kbase_va_region *kbase_tmem_alloc(struct kbase_context *kctx,
+ u32 vsize, u32 psize,
+ u32 extent, u32 flags, mali_bool is_growable);
+
+/** Resize a tmem region
+ *
+ * This function changes the number of physical pages committed to a tmem region.
+ *
+ * @param[in] kctx The kbase context which the tmem belongs to
+ * @param[in] gpu_addr The base address of the tmem region
+ * @param[in] delta The number of pages to grow or shrink by
+ * @param[out] size The number of pages of memory committed after growing/shrinking
+ * @param[out] failure_reason Error code describing reason of failure.
+ *
+ * @return MALI_ERROR_NONE on success
+ */
+mali_error kbase_tmem_resize(struct kbase_context *kctx, mali_addr64 gpu_addr, s32 delta, u32 *size, base_backing_threshold_status * failure_reason);
+
+/**
+ * Import external memory.
+ *
+ * This function supports importing external memory.
+ * If imported a kbase_va_region is created of the tmem type.
+ * The region might not be mappable on the CPU depending on the imported type.
+ * If not mappable the KBASE_REG_NO_CPU_MAP bit will be set.
+ *
+ * Import will fail if (but not limited to):
+ * @li Unsupported import type
+ * @li Handle not valid for the type
+ * @li Access to a handle was not valid
+ * @li The underlying memory can't be accessed by the GPU
+ * @li No VA space found to map the memory
+ * @li Resources to track the region was not available
+ *
+ * @param[in] kctx The kbase context which the tmem will be created in
+ * @param type The type of memory to import
+ * @param handle Handle to the memory to import
+ * @param[out] pages Where to store the number of pages imported
+ * @return A region pointer on success, NULL on failure
+ */
+struct kbase_va_region *kbase_tmem_import(struct kbase_context *kctx, base_tmem_import_type type, int handle, u64 * const pages);
+
+
+/* OS specific functions */
+struct kbase_va_region * kbase_lookup_cookie(struct kbase_context * kctx, mali_addr64 cookie);
+void kbase_unlink_cookie(struct kbase_context * kctx, mali_addr64 cookie, struct kbase_va_region * reg);
+mali_error kbase_mem_free(struct kbase_context *kctx, mali_addr64 gpu_addr);
+mali_error kbase_mem_free_region(struct kbase_context *kctx,
+ struct kbase_va_region *reg);
+void kbase_os_mem_map_lock(struct kbase_context * kctx);
+void kbase_os_mem_map_unlock(struct kbase_context * kctx);
+
+/**
+ * @brief Find a CPU mapping of a memory allocation containing a given address range
+ *
+ * Searches for a CPU mapping of any part of the region starting at @p gpu_addr that
+ * fully encloses the CPU virtual address range specified by @p uaddr and @p size.
+ * Returns a failure indication if only part of the address range lies within a
+ * CPU mapping, or the address range lies within a CPU mapping of a different region.
+ *
+ * @param[in,out] kctx The kernel base context used for the allocation.
+ * @param[in] gpu_addr GPU address of the start of the allocated region
+ * within which to search.
+ * @param[in] uaddr Start of the CPU virtual address range.
+ * @param[in] size Size of the CPU virtual address range (in bytes).
+ *
+ * @return A pointer to a descriptor of the CPU mapping that fully encloses
+ * the specified address range, or NULL if none was found.
+ */
+struct kbase_cpu_mapping *kbasep_find_enclosing_cpu_mapping(
+ struct kbase_context *kctx,
+ mali_addr64 gpu_addr,
+ osk_virt_addr uaddr,
+ size_t size );
+
+/**
+ * @brief Round TMem Growable no. pages to allow for HW workarounds/block allocators
+ *
+ * For success, the caller should check that the unsigned return value is
+ * higher than the \a nr_pages parameter.
+ *
+ * @param[in] nr_pages Size value (in pages) to round
+ * @return the rounded-up number of pages (which may have wraped around to zero)
+ */
+static INLINE u32 kbasep_tmem_growable_round_size( kbase_device *kbdev, u32 nr_pages )
+{
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_9630))
+ {
+ return (nr_pages + KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_HW_ISSUE_9630 - 1) & ~(KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_HW_ISSUE_9630 - 1);
+ }
+ else if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8316))
+ {
+ return (nr_pages + KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_HW_ISSUE_8316 - 1) & ~(KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES_HW_ISSUE_8316 - 1);
+ }
+ else
+ {
+ return (nr_pages + KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES - 1) & ~(KBASEP_TMEM_GROWABLE_BLOCKSIZE_PAGES-1);
+ }
+}
+
+void kbasep_as_poke_timer_callback(void* arg);
+void kbase_as_poking_timer_retain(kbase_as * as);
+void kbase_as_poking_timer_release(kbase_as * as);
+
+
+#endif /* _KBASE_MEM_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_mmu.c
+ * Base kernel MMU management.
+ */
+
+/* #define DEBUG 1 */
+#include <osk/mali_osk.h>
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_midg_regmap.h>
+#include <kbase/src/common/mali_kbase_gator.h>
+
+#define beenthere(f, a...) OSK_PRINT_INFO(OSK_BASE_MMU, "%s:" f, __func__, ##a)
+
+#include <kbase/src/common/mali_kbase_defs.h>
+#include <kbase/src/common/mali_kbase_hw.h>
+
+#define KBASE_MMU_PAGE_ENTRIES 512
+
+/*
+ * Definitions:
+ * - PGD: Page Directory.
+ * - PTE: Page Table Entry. A 64bit value pointing to the next
+ * level of translation
+ * - ATE: Address Transation Entry. A 64bit value pointing to
+ * a 4kB physical page.
+ */
+
+static void kbase_mmu_report_fault_and_kill(kbase_context *kctx, kbase_as * as, mali_addr64 fault_addr);
+static u64 lock_region(kbase_device * kbdev, u64 pfn, u32 num_pages);
+
+static void ksync_kern_vrange_gpu(osk_phy_addr paddr, osk_virt_addr vaddr, size_t size)
+{
+ osk_sync_to_memory(paddr, vaddr, size);
+}
+
+static u32 make_multiple(u32 minimum, u32 multiple)
+{
+ u32 remainder = minimum % multiple;
+ if (remainder == 0)
+ {
+ return minimum;
+ }
+ else
+ {
+ return minimum + multiple - remainder;
+ }
+}
+
+static void mmu_mask_reenable(kbase_device * kbdev, kbase_context *kctx, kbase_as * as)
+{
+ u32 mask;
+ osk_spinlock_irq_lock(&kbdev->mmu_mask_change);
+ mask = kbase_reg_read(kbdev, MMU_REG(MMU_IRQ_MASK), kctx);
+ mask |= ((1UL << as->number) | (1UL << (MMU_REGS_BUS_ERROR_FLAG(as->number))));
+ kbase_reg_write(kbdev, MMU_REG(MMU_IRQ_MASK), mask, kctx);
+ osk_spinlock_irq_unlock(&kbdev->mmu_mask_change);
+}
+
+static void page_fault_worker(osk_workq_work *data)
+{
+ u64 fault_pfn;
+ u32 new_pages;
+ u32 fault_rel_pfn;
+ kbase_as * faulting_as;
+ int as_no;
+ kbase_context * kctx;
+ kbase_device * kbdev;
+ kbase_va_region *region;
+ mali_error err;
+
+ u32 fault_status;
+
+ faulting_as = CONTAINER_OF(data, kbase_as, work_pagefault);
+ fault_pfn = faulting_as->fault_addr >> OSK_PAGE_SHIFT;
+ as_no = faulting_as->number;
+
+ kbdev = CONTAINER_OF( faulting_as, kbase_device, as[as_no] );
+
+ /* Grab the context that was already refcounted in kbase_mmu_interrupt().
+ * Therefore, it cannot be scheduled out of this AS until we explicitly release it
+ *
+ * NOTE: NULL can be returned here if we're gracefully handling a spurious interrupt */
+ kctx = kbasep_js_runpool_lookup_ctx_noretain( kbdev, as_no );
+
+ if ( kctx == NULL )
+ {
+ /* Address space has no context, terminate the work */
+ u32 reg;
+ /* AS transaction begin */
+ osk_mutex_lock(&faulting_as->transaction_mutex);
+ reg = kbase_reg_read(kbdev, MMU_AS_REG(as_no, ASn_TRANSTAB_LO), NULL);
+ reg = (reg & (~(u32)MMU_TRANSTAB_ADRMODE_MASK)) | ASn_TRANSTAB_ADRMODE_UNMAPPED;
+ kbase_reg_write(kbdev, MMU_AS_REG(as_no, ASn_TRANSTAB_LO), reg, NULL);
+ kbase_reg_write(kbdev, MMU_AS_REG(as_no, ASn_COMMAND), ASn_COMMAND_UPDATE, NULL);
+ kbase_reg_write(kbdev, MMU_REG(MMU_IRQ_CLEAR), (1UL << as_no), NULL);
+ osk_mutex_unlock(&faulting_as->transaction_mutex);
+ /* AS transaction end */
+
+ mmu_mask_reenable(kbdev, NULL, faulting_as);
+ return;
+ }
+
+ fault_status = kbase_reg_read(kbdev, MMU_AS_REG(as_no, ASn_FAULTSTATUS), NULL);
+
+ OSK_ASSERT( kctx->kbdev == kbdev );
+
+ kbase_gpu_vm_lock(kctx);
+
+ /* find the region object for this VA */
+ region = kbase_region_lookup(kctx, faulting_as->fault_addr);
+ if (NULL == region || (GROWABLE_FLAGS_REQUIRED != (region->flags & GROWABLE_FLAGS_MASK)))
+ {
+ kbase_gpu_vm_unlock(kctx);
+ /* failed to find the region or mismatch of the flags */
+ kbase_mmu_report_fault_and_kill(kctx, faulting_as, faulting_as->fault_addr);
+ goto fault_done;
+ }
+
+ if ((((fault_status & ASn_FAULTSTATUS_ACCESS_TYPE_MASK) == ASn_FAULTSTATUS_ACCESS_TYPE_READ) &&
+ !(region->flags & KBASE_REG_GPU_RD)) ||
+ (((fault_status & ASn_FAULTSTATUS_ACCESS_TYPE_MASK) == ASn_FAULTSTATUS_ACCESS_TYPE_WRITE) &&
+ !(region->flags & KBASE_REG_GPU_WR)) ||
+ (((fault_status & ASn_FAULTSTATUS_ACCESS_TYPE_MASK) == ASn_FAULTSTATUS_ACCESS_TYPE_EX) &&
+ (region->flags & KBASE_REG_GPU_NX)))
+ {
+ OSK_PRINT_WARN(OSK_BASE_MMU, "Access permissions don't match: region->flags=0x%x", region->flags);
+ kbase_gpu_vm_unlock(kctx);
+ kbase_mmu_report_fault_and_kill(kctx, faulting_as, faulting_as->fault_addr);
+ goto fault_done;
+ }
+
+ /* find the size we need to grow it by */
+ /* we know the result fit in a u32 due to kbase_region_lookup
+ * validating the fault_adress to be within a u32 from the start_pfn */
+ fault_rel_pfn = fault_pfn - region->start_pfn;
+
+ if (fault_rel_pfn < region->nr_alloc_pages)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MMU, "Fault in allocated region of growable TMEM: Ignoring");
+ kbase_reg_write(kbdev, MMU_REG(MMU_IRQ_CLEAR), (1UL << as_no), NULL);
+ mmu_mask_reenable(kbdev, kctx, faulting_as);
+ kbase_gpu_vm_unlock(kctx);
+ goto fault_done;
+ }
+
+ new_pages = make_multiple(fault_rel_pfn - region->nr_alloc_pages + 1, region->extent);
+ if (new_pages + region->nr_alloc_pages > region->nr_pages)
+ {
+ /* cap to max vsize */
+ new_pages = region->nr_pages - region->nr_alloc_pages;
+ }
+
+ if (0 == new_pages)
+ {
+ /* Duplicate of a fault we've already handled, nothing to do */
+ kbase_reg_write(kbdev, MMU_REG(MMU_IRQ_CLEAR), (1UL << as_no), NULL);
+ mmu_mask_reenable(kbdev, kctx, faulting_as);
+ kbase_gpu_vm_unlock(kctx);
+ goto fault_done;
+ }
+
+ if (MALI_ERROR_NONE == kbase_alloc_phy_pages_helper(region, new_pages))
+ {
+ /* alloc success */
+ mali_addr64 lock_addr;
+ OSK_ASSERT(region->nr_alloc_pages <= region->nr_pages);
+
+ /* AS transaction begin */
+ osk_mutex_lock(&faulting_as->transaction_mutex);
+
+ /* Lock the VA region we're about to update */
+ lock_addr = lock_region(kbdev, faulting_as->fault_addr >> OSK_PAGE_SHIFT, new_pages);
+ kbase_reg_write(kbdev, MMU_AS_REG(as_no, ASn_LOCKADDR_LO), lock_addr & 0xFFFFFFFFUL, kctx);
+ kbase_reg_write(kbdev, MMU_AS_REG(as_no, ASn_LOCKADDR_HI), lock_addr >> 32, kctx);
+ kbase_reg_write(kbdev, MMU_AS_REG(as_no, ASn_COMMAND), ASn_COMMAND_LOCK, kctx);
+
+ /* set up the new pages */
+ err = kbase_mmu_insert_pages(kctx, region->start_pfn + region->nr_alloc_pages - new_pages,
+ ®ion->phy_pages[region->nr_alloc_pages - new_pages],
+ new_pages, region->flags);
+ if(MALI_ERROR_NONE != err)
+ {
+ /* failed to insert pages, handle as a normal PF */
+ osk_mutex_unlock(&faulting_as->transaction_mutex);
+ kbase_gpu_vm_unlock(kctx);
+ /* The locked VA region will be unlocked and the cache invalidated in here */
+ kbase_mmu_report_fault_and_kill(kctx, faulting_as, faulting_as->fault_addr);
+ goto fault_done;
+ }
+
+#if MALI_GATOR_SUPPORT
+ kbase_trace_mali_page_fault_insert_pages(as_no, new_pages);
+#endif
+ /* clear the irq */
+ /* MUST BE BEFORE THE FLUSH/UNLOCK */
+ kbase_reg_write(kbdev, MMU_REG(MMU_IRQ_CLEAR), (1UL << as_no), NULL);
+
+ /* flush L2 and unlock the VA (resumes the MMU) */
+ kbase_reg_write(kbdev, MMU_AS_REG(as_no, ASn_COMMAND), ASn_COMMAND_FLUSH, kctx);
+
+ /* wait for the flush to complete */
+ while (kbase_reg_read(kbdev, MMU_AS_REG(as_no, ASn_STATUS), kctx) & 1);
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_9630))
+ {
+ /* Issue an UNLOCK command to ensure that valid page tables are re-read by the GPU after an update.
+ Note that, the FLUSH command should perform all the actions necessary, however the bus logs show
+ that if multiple page faults occur within an 8 page region the MMU does not always re-read the
+ updated page table entries for later faults or is only partially read, it subsequently raises the
+ page fault IRQ for the same addresses, the unlock ensures that the MMU cache is flushed, so updates
+ can be re-read. As the region is now unlocked we need to issue 2 UNLOCK commands in order to flush the
+ MMU/uTLB, see PRLAM-8812.
+ */
+ kbase_reg_write(kctx->kbdev, MMU_AS_REG(kctx->as_nr, ASn_COMMAND), ASn_COMMAND_UNLOCK, kctx);
+ kbase_reg_write(kctx->kbdev, MMU_AS_REG(kctx->as_nr, ASn_COMMAND), ASn_COMMAND_UNLOCK, kctx);
+ }
+
+ osk_mutex_unlock(&faulting_as->transaction_mutex);
+ /* AS transaction end */
+
+ /* reenable this in the mask */
+ mmu_mask_reenable(kbdev, kctx, faulting_as);
+ kbase_gpu_vm_unlock(kctx);
+ }
+ else
+ {
+ /* failed to extend, handle as a normal PF */
+ kbase_gpu_vm_unlock(kctx);
+ kbase_mmu_report_fault_and_kill(kctx, faulting_as, faulting_as->fault_addr);
+ }
+
+
+fault_done:
+ /* By this point, the fault was handled in some way, so release the ctx refcount */
+ kbasep_js_runpool_release_ctx( kbdev, kctx );
+}
+
+osk_phy_addr kbase_mmu_alloc_pgd(kbase_context *kctx)
+{
+ osk_phy_addr pgd;
+ u64 *page;
+ int i;
+ u32 count;
+ OSK_ASSERT( NULL != kctx);
+ if (MALI_ERROR_NONE != kbase_mem_usage_request_pages(&kctx->usage, 1))
+ {
+ return 0;
+ }
+
+ count = kbase_phy_pages_alloc(kctx->kbdev, &kctx->pgd_allocator, 1, &pgd);
+ if (count != 1)
+ {
+ kbase_mem_usage_release_pages(&kctx->usage, 1);
+ return 0;
+ }
+
+ page = osk_kmap(pgd);
+ if(NULL == page)
+ {
+ kbase_phy_pages_free(kctx->kbdev, &kctx->pgd_allocator, 1, &pgd);
+ kbase_mem_usage_release_pages(&kctx->usage, 1);
+ return 0;
+ }
+
+ for (i = 0; i < KBASE_MMU_PAGE_ENTRIES; i++)
+ page[i] = ENTRY_IS_INVAL;
+
+ /* Clean the full page */
+ ksync_kern_vrange_gpu(pgd, page, KBASE_MMU_PAGE_ENTRIES * sizeof(u64));
+ osk_kunmap(pgd, page);
+ return pgd;
+}
+KBASE_EXPORT_TEST_API(kbase_mmu_alloc_pgd)
+
+static osk_phy_addr mmu_pte_to_phy_addr(u64 entry)
+{
+ if (!(entry & 1))
+ return 0;
+
+ return entry & ~0xFFF;
+}
+
+static u64 mmu_phyaddr_to_pte(osk_phy_addr phy)
+{
+ return (phy & ~0xFFF) | ENTRY_IS_PTE;
+}
+
+static u64 mmu_phyaddr_to_ate(osk_phy_addr phy, u64 flags)
+{
+ return (phy & ~0xFFF) | (flags & ENTRY_FLAGS_MASK) | ENTRY_IS_ATE;
+}
+
+/* Given PGD PFN for level N, return PGD PFN for level N+1 */
+static osk_phy_addr mmu_get_next_pgd(struct kbase_context *kctx,
+ osk_phy_addr pgd, u64 vpfn, int level)
+{
+ u64 *page;
+ osk_phy_addr target_pgd;
+
+ OSK_ASSERT(pgd);
+
+ /*
+ * Architecture spec defines level-0 as being the top-most.
+ * This is a bit unfortunate here, but we keep the same convention.
+ */
+ vpfn >>= (3 - level) * 9;
+ vpfn &= 0x1FF;
+
+ page = osk_kmap(pgd);
+ if(NULL == page)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MMU, "mmu_get_next_pgd: kmap failure\n");
+ return 0;
+ }
+
+ target_pgd = mmu_pte_to_phy_addr(page[vpfn]);
+
+ if (!target_pgd) {
+ target_pgd = kbase_mmu_alloc_pgd(kctx);
+ if(!target_pgd)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MMU, "mmu_get_next_pgd: kbase_mmu_alloc_pgd failure\n");
+ osk_kunmap(pgd, page);
+ return 0;
+ }
+
+ page[vpfn] = mmu_phyaddr_to_pte(target_pgd);
+ ksync_kern_vrange_gpu(pgd + (vpfn * sizeof(u64)), page + vpfn, sizeof(u64));
+ /* Rely on the caller to update the address space flags. */
+ }
+
+ osk_kunmap(pgd, page);
+ return target_pgd;
+}
+
+static osk_phy_addr mmu_get_bottom_pgd(struct kbase_context *kctx, u64 vpfn)
+{
+ osk_phy_addr pgd;
+ int l;
+
+ pgd = kctx->pgd;
+
+ for (l = MIDGARD_MMU_TOPLEVEL; l < 3; l++) {
+ pgd = mmu_get_next_pgd(kctx, pgd, vpfn, l);
+ /* Handle failure condition */
+ if(!pgd)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MMU, "mmu_get_bottom_pgd: mmu_get_next_pgd failure\n");
+ return 0;
+ }
+ }
+
+ return pgd;
+}
+
+static osk_phy_addr mmu_insert_pages_recover_get_next_pgd(struct kbase_context *kctx,
+ osk_phy_addr pgd, u64 vpfn, int level)
+{
+ u64 *page;
+ osk_phy_addr target_pgd;
+
+ OSK_ASSERT(pgd);
+ CSTD_UNUSED(kctx);
+
+ /*
+ * Architecture spec defines level-0 as being the top-most.
+ * This is a bit unfortunate here, but we keep the same convention.
+ */
+ vpfn >>= (3 - level) * 9;
+ vpfn &= 0x1FF;
+
+ page = osk_kmap_atomic(pgd);
+ /* osk_kmap_atomic should NEVER fail */
+ OSK_ASSERT(NULL != page);
+
+ target_pgd = mmu_pte_to_phy_addr(page[vpfn]);
+ /* As we are recovering from what has already been set up, we should have a target_pgd */
+ OSK_ASSERT(0 != target_pgd);
+
+ osk_kunmap_atomic(pgd, page);
+ return target_pgd;
+}
+
+static osk_phy_addr mmu_insert_pages_recover_get_bottom_pgd(struct kbase_context *kctx, u64 vpfn)
+{
+ osk_phy_addr pgd;
+ int l;
+
+ pgd = kctx->pgd;
+
+ for (l = MIDGARD_MMU_TOPLEVEL; l < 3; l++) {
+ pgd = mmu_insert_pages_recover_get_next_pgd(kctx, pgd, vpfn, l);
+ /* Should never fail */
+ OSK_ASSERT(0 != pgd);
+ }
+
+ return pgd;
+}
+
+static void mmu_insert_pages_failure_recovery(struct kbase_context *kctx, u64 vpfn,
+ osk_phy_addr *phys, u32 nr)
+{
+ osk_phy_addr pgd;
+ u64 *pgd_page;
+
+ OSK_ASSERT( NULL != kctx );
+ OSK_ASSERT( 0 != vpfn );
+ OSK_ASSERT( vpfn <= (UINT64_MAX / OSK_PAGE_SIZE) ); /* 64-bit address range is the max */
+
+ while (nr) {
+ u32 i;
+ u32 index = vpfn & 0x1FF;
+ u32 count = KBASE_MMU_PAGE_ENTRIES - index;
+
+ if (count > nr)
+ {
+ count = nr;
+ }
+
+ pgd = mmu_insert_pages_recover_get_bottom_pgd(kctx, vpfn);
+ OSK_ASSERT(0 != pgd);
+
+ pgd_page = osk_kmap_atomic(pgd);
+ OSK_ASSERT(NULL != pgd_page);
+
+ /* Invalidate the entries we added */
+ for (i = 0; i < count; i++) {
+ pgd_page[index + i] = ENTRY_IS_INVAL;
+ }
+
+ phys += count;
+ vpfn += count;
+ nr -= count;
+
+ ksync_kern_vrange_gpu(pgd + (index * sizeof(u64)), pgd_page + index, count * sizeof(u64));
+
+ osk_kunmap_atomic(pgd, pgd_page);
+ }
+}
+
+/*
+ * Map 'nr' pages pointed to by 'phys' at GPU PFN 'vpfn'
+ */
+mali_error kbase_mmu_insert_pages(struct kbase_context *kctx, u64 vpfn,
+ osk_phy_addr *phys, u32 nr, u32 flags)
+{
+ osk_phy_addr pgd;
+ u64 *pgd_page;
+ u64 mmu_flags = 0;
+ /* In case the insert_pages only partially completes we need to be able to recover */
+ mali_bool recover_required = MALI_FALSE;
+ u64 recover_vpfn = vpfn;
+ osk_phy_addr *recover_phys = phys;
+ u32 recover_count = 0;
+
+ OSK_ASSERT( NULL != kctx );
+ OSK_ASSERT( 0 != vpfn );
+ OSK_ASSERT( (flags & ~((1 << KBASE_REG_FLAGS_NR_BITS) - 1)) == 0 );
+ OSK_ASSERT( vpfn <= (UINT64_MAX / OSK_PAGE_SIZE) ); /* 64-bit address range is the max */
+
+ mmu_flags |= (flags & KBASE_REG_GPU_WR) ? ENTRY_WR_BIT : 0; /* write perm if requested */
+ mmu_flags |= (flags & KBASE_REG_GPU_RD) ? ENTRY_RD_BIT : 0; /* read perm if requested */
+ mmu_flags |= (flags & KBASE_REG_GPU_NX) ? ENTRY_NX_BIT : 0; /* nx if requested */
+
+ if (flags & KBASE_REG_SHARE_BOTH)
+ {
+ /* inner and outer shareable */
+ mmu_flags |= SHARE_BOTH_BITS;
+ }
+ else if (flags & KBASE_REG_SHARE_IN)
+ {
+ /* inner shareable coherency */
+ mmu_flags |= SHARE_INNER_BITS;
+ }
+
+ while (nr) {
+ u32 i;
+ u32 index = vpfn & 0x1FF;
+ u32 count = KBASE_MMU_PAGE_ENTRIES - index;
+
+ if (count > nr)
+ count = nr;
+
+ /*
+ * Repeatedly calling mmu_get_bottom_pte() is clearly
+ * suboptimal. We don't have to re-parse the whole tree
+ * each time (just cache the l0-l2 sequence).
+ * On the other hand, it's only a gain when we map more than
+ * 256 pages at once (on average). Do we really care?
+ */
+ pgd = mmu_get_bottom_pgd(kctx, vpfn);
+ if(!pgd)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MMU, "kbase_mmu_insert_pages: mmu_get_bottom_pgd failure\n");
+ if(recover_required)
+ {
+ /* Invalidate the pages we have partially completed */
+ mmu_insert_pages_failure_recovery(kctx, recover_vpfn, recover_phys, recover_count);
+ }
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ pgd_page = osk_kmap(pgd);
+ if(!pgd_page)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MMU, "kbase_mmu_insert_pages: kmap failure\n");
+ if(recover_required)
+ {
+ /* Invalidate the pages we have partially completed */
+ mmu_insert_pages_failure_recovery(kctx, recover_vpfn, recover_phys, recover_count);
+ }
+ return MALI_ERROR_OUT_OF_MEMORY;
+ }
+
+ for (i = 0; i < count; i++) {
+ OSK_ASSERT(0 == (pgd_page[index + i] & 1UL));
+ pgd_page[index + i] = mmu_phyaddr_to_ate(phys[i], mmu_flags);
+ }
+
+ phys += count;
+ vpfn += count;
+ nr -= count;
+
+ ksync_kern_vrange_gpu(pgd + (index * sizeof(u64)), pgd_page + index, count * sizeof(u64));
+
+ osk_kunmap(pgd, pgd_page);
+ /* We have started modifying the page table. If further pages need inserting and fail we need to
+ * undo what has already taken place */
+ recover_required = MALI_TRUE;
+ recover_count+= count;
+ }
+ return MALI_ERROR_NONE;
+}
+KBASE_EXPORT_TEST_API(kbase_mmu_insert_pages)
+
+/*
+ * We actually only discard the ATE, and not the page table
+ * pages. There is a potential DoS here, as we'll leak memory by
+ * having PTEs that are potentially unused. Will require physical
+ * page accounting, so MMU pages are part of the process allocation.
+ *
+ * IMPORTANT: This uses kbasep_js_runpool_release_ctx() when the context is
+ * currently scheduled into the runpool, and so potentially uses a lot of locks.
+ * These locks must be taken in the correct order with respect to others
+ * already held by the caller. Refer to kbasep_js_runpool_release_ctx() for more
+ * information.
+ */
+ mali_error kbase_mmu_teardown_pages(struct kbase_context *kctx, u64 vpfn, u32 nr)
+{
+ osk_phy_addr pgd;
+ u64 *pgd_page;
+ kbase_device *kbdev;
+ mali_bool ctx_is_in_runpool;
+ u32 requested_nr = nr;
+
+ beenthere("kctx %p vpfn %lx nr %d", (void *)kctx, (unsigned long)vpfn, nr);
+
+ OSK_ASSERT(NULL != kctx);
+
+ if (0 == nr)
+ {
+ /* early out if nothing to do */
+ return MALI_ERROR_NONE;
+ }
+
+ kbdev = kctx->kbdev;
+
+ while (nr)
+ {
+ u32 i;
+ u32 index = vpfn & 0x1FF;
+ u32 count = KBASE_MMU_PAGE_ENTRIES - index;
+ if (count > nr)
+ count = nr;
+
+ pgd = mmu_get_bottom_pgd(kctx, vpfn);
+ if(!pgd)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MMU, "kbase_mmu_teardown_pages: mmu_get_bottom_pgd failure\n");
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ pgd_page = osk_kmap(pgd);
+ if(!pgd_page)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MMU, "kbase_mmu_teardown_pages: kmap failure\n");
+ return MALI_ERROR_OUT_OF_MEMORY;
+ }
+
+ for (i = 0; i < count; i++) {
+ /*
+ * Possible micro-optimisation: only write to the
+ * low 32bits. That's enough to invalidate the mapping.
+ */
+ pgd_page[index + i] = ENTRY_IS_INVAL;
+ }
+
+ vpfn += count;
+ nr -= count;
+
+ ksync_kern_vrange_gpu(pgd + (index * sizeof(u64)), pgd_page + index, count * sizeof(u64));
+
+ osk_kunmap(pgd, pgd_page);
+ }
+
+ /* We must flush if we're currently running jobs. At the very least, we need to retain the
+ * context to ensure it doesn't schedule out whilst we're trying to flush it */
+ ctx_is_in_runpool = kbasep_js_runpool_retain_ctx( kbdev, kctx );
+
+ if ( ctx_is_in_runpool )
+ {
+ OSK_ASSERT( kctx->as_nr != KBASEP_AS_NR_INVALID );
+
+ /* Second level check is to try to only do this when jobs are running. The refcount is
+ * a heuristic for this. */
+ if ( kbdev->js_data.runpool_irq.per_as_data[kctx->as_nr].as_busy_refcount >= 2 )
+ {
+ /* Lock the VA region we're about to update */
+ u64 lock_addr = lock_region(kbdev, vpfn, requested_nr);
+
+ /* AS transaction begin */
+ osk_mutex_lock(&kbdev->as[kctx->as_nr].transaction_mutex);
+ kbase_reg_write(kctx->kbdev, MMU_AS_REG(kctx->as_nr, ASn_LOCKADDR_LO), lock_addr & 0xFFFFFFFFUL, kctx);
+ kbase_reg_write(kctx->kbdev, MMU_AS_REG(kctx->as_nr, ASn_LOCKADDR_HI), lock_addr >> 32, kctx);
+ kbase_reg_write(kctx->kbdev, MMU_AS_REG(kctx->as_nr, ASn_COMMAND), ASn_COMMAND_LOCK, kctx);
+
+ /* flush L2 and unlock the VA */
+ kbase_reg_write(kctx->kbdev, MMU_AS_REG(kctx->as_nr, ASn_COMMAND), ASn_COMMAND_FLUSH, kctx);
+
+ /* wait for the flush to complete */
+ while (kbase_reg_read(kctx->kbdev, MMU_AS_REG(kctx->as_nr, ASn_STATUS), kctx) & ASn_STATUS_FLUSH_ACTIVE);
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_9630))
+ {
+ /* Issue an UNLOCK command to ensure that valid page tables are re-read by the GPU after an update.
+ Note that, the FLUSH command should perform all the actions necessary, however the bus logs show
+ that if multiple page faults occur within an 8 page region the MMU does not always re-read the
+ updated page table entries for later faults or is only partially read, it subsequently raises the
+ page fault IRQ for the same addresses, the unlock ensures that the MMU cache is flushed, so updates
+ can be re-read. As the region is now unlocked we need to issue 2 UNLOCK commands in order to flush the
+ MMU/uTLB, see PRLAM-8812.
+ */
+ kbase_reg_write(kctx->kbdev, MMU_AS_REG(kctx->as_nr, ASn_COMMAND), ASn_COMMAND_UNLOCK, kctx);
+ kbase_reg_write(kctx->kbdev, MMU_AS_REG(kctx->as_nr, ASn_COMMAND), ASn_COMMAND_UNLOCK, kctx);
+ }
+
+ osk_mutex_unlock(&kbdev->as[kctx->as_nr].transaction_mutex);
+ /* AS transaction end */
+ }
+ kbasep_js_runpool_release_ctx( kbdev, kctx );
+ }
+ return MALI_ERROR_NONE;
+}
+KBASE_EXPORT_TEST_API(kbase_mmu_teardown_pages)
+
+static int mmu_pte_is_valid(u64 pte)
+{
+ return ((pte & 3) == ENTRY_IS_ATE);
+}
+
+/* This is a debug feature only */
+static void mmu_check_unused(kbase_context *kctx, osk_phy_addr pgd)
+{
+ u64 *page;
+ int i;
+ CSTD_UNUSED(kctx);
+
+ page = osk_kmap_atomic(pgd);
+ /* kmap_atomic should NEVER fail. */
+ OSK_ASSERT(NULL != page);
+
+ for (i = 0; i < KBASE_MMU_PAGE_ENTRIES; i++)
+ {
+ if (mmu_pte_is_valid(page[i]))
+ {
+ beenthere("live pte %016lx", (unsigned long)page[i]);
+ }
+ }
+ osk_kunmap_atomic(pgd, page);
+}
+
+static void mmu_teardown_level(kbase_context *kctx, osk_phy_addr pgd, int level, int zap, u64 *pgd_page_buffer)
+{
+ osk_phy_addr target_pgd;
+ u64 *pgd_page;
+ int i;
+
+ pgd_page = osk_kmap_atomic(pgd);
+ /* kmap_atomic should NEVER fail. */
+ OSK_ASSERT(NULL != pgd_page);
+ /* Copy the page to our preallocated buffer so that we can minimize kmap_atomic usage */
+ memcpy(pgd_page_buffer, pgd_page, OSK_PAGE_SIZE);
+ osk_kunmap_atomic(pgd, pgd_page);
+ pgd_page = pgd_page_buffer;
+
+ for (i = 0; i < KBASE_MMU_PAGE_ENTRIES; i++) {
+ target_pgd = mmu_pte_to_phy_addr(pgd_page[i]);
+
+ if (target_pgd) {
+ if (level < 2)
+ {
+ mmu_teardown_level(kctx, target_pgd, level + 1, zap, pgd_page_buffer+(OSK_PAGE_SIZE/sizeof(u64)));
+ }
+ else {
+ /*
+ * So target_pte is a level-3 page.
+ * As a leaf, it is safe to free it.
+ * Unless we have live pages attached to it!
+ */
+ mmu_check_unused(kctx, target_pgd);
+ }
+
+ beenthere("pte %lx level %d", (unsigned long)target_pgd, level + 1);
+ if (zap)
+ {
+ kbase_phy_pages_free(kctx->kbdev, &kctx->pgd_allocator, 1, &target_pgd);
+ kbase_mem_usage_release_pages(&kctx->usage, 1);
+ }
+ }
+ }
+}
+
+mali_error kbase_mmu_init(struct kbase_context *kctx)
+{
+ OSK_ASSERT(NULL != kctx);
+ OSK_ASSERT(NULL == kctx->mmu_teardown_pages);
+
+ /* Preallocate MMU depth of four pages for mmu_teardown_level to use */
+ kctx->mmu_teardown_pages = osk_malloc(OSK_PAGE_SIZE*4);
+ if(NULL == kctx->mmu_teardown_pages)
+ {
+ return MALI_ERROR_OUT_OF_MEMORY;
+ }
+ return MALI_ERROR_NONE;
+}
+
+void kbase_mmu_term(struct kbase_context *kctx)
+{
+ OSK_ASSERT(NULL != kctx);
+ OSK_ASSERT(NULL != kctx->mmu_teardown_pages);
+
+ osk_free(kctx->mmu_teardown_pages);
+ kctx->mmu_teardown_pages = NULL;
+}
+
+void kbase_mmu_free_pgd(struct kbase_context *kctx)
+{
+ OSK_ASSERT(NULL != kctx);
+ OSK_ASSERT(NULL != kctx->mmu_teardown_pages);
+
+ mmu_teardown_level(kctx, kctx->pgd, MIDGARD_MMU_TOPLEVEL, 1, kctx->mmu_teardown_pages);
+
+ beenthere("pgd %lx", (unsigned long)kctx->pgd);
+ kbase_phy_pages_free(kctx->kbdev, &kctx->pgd_allocator, 1, &kctx->pgd);
+ kbase_mem_usage_release_pages(&kctx->usage, 1);
+}
+KBASE_EXPORT_TEST_API(kbase_mmu_free_pgd)
+
+static size_t kbasep_mmu_dump_level(kbase_context *kctx, osk_phy_addr pgd, int level, char **buffer, size_t *size_left)
+{
+ osk_phy_addr target_pgd;
+ u64 *pgd_page;
+ int i;
+ size_t size = KBASE_MMU_PAGE_ENTRIES*sizeof(u64)+sizeof(u64);
+ size_t dump_size;
+
+ pgd_page = osk_kmap(pgd);
+ if(!pgd_page)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MMU, "kbasep_mmu_dump_level: kmap failure\n");
+ return 0;
+ }
+
+ if (*size_left >= size)
+ {
+ /* A modified physical address that contains the page table level */
+ u64 m_pgd = pgd | level;
+
+ /* Put the modified physical address in the output buffer */
+ memcpy(*buffer, &m_pgd, sizeof(m_pgd));
+ *buffer += sizeof(m_pgd);
+
+ /* Followed by the page table itself */
+ memcpy(*buffer, pgd_page, sizeof(u64)*KBASE_MMU_PAGE_ENTRIES);
+ *buffer += sizeof(u64)*KBASE_MMU_PAGE_ENTRIES;
+
+ *size_left -= size;
+ }
+
+ for (i = 0; i < KBASE_MMU_PAGE_ENTRIES; i++) {
+ if ((pgd_page[i] & ENTRY_IS_PTE) == ENTRY_IS_PTE) {
+ target_pgd = mmu_pte_to_phy_addr(pgd_page[i]);
+
+ dump_size = kbasep_mmu_dump_level(kctx, target_pgd, level + 1, buffer, size_left);
+ if(!dump_size)
+ {
+ osk_kunmap(pgd, pgd_page);
+ return 0;
+ }
+ size += dump_size;
+ }
+ }
+
+ osk_kunmap(pgd, pgd_page);
+
+ return size;
+}
+
+void *kbase_mmu_dump(struct kbase_context *kctx,int nr_pages)
+{
+ void *kaddr;
+ size_t size_left;
+
+ OSK_ASSERT(kctx);
+
+ if (0 == nr_pages)
+ {
+ /* can't find in a 0 sized buffer, early out */
+ return NULL;
+ }
+
+ size_left = nr_pages * OSK_PAGE_SIZE;
+
+ kaddr = osk_vmalloc(size_left);
+
+ if (kaddr)
+ {
+ u64 end_marker = 0xFFULL;
+ char *buffer = (char*)kaddr;
+
+ size_t size = kbasep_mmu_dump_level(kctx, kctx->pgd, MIDGARD_MMU_TOPLEVEL, &buffer, &size_left);
+ if(!size)
+ {
+ osk_vfree(kaddr);
+ return NULL;
+ }
+
+ /* Add on the size for the end marker */
+ size += sizeof(u64);
+
+ if (size > nr_pages * OSK_PAGE_SIZE || size_left < sizeof(u64)) {
+ /* The buffer isn't big enough - free the memory and return failure */
+ osk_vfree(kaddr);
+ return NULL;
+ }
+
+ /* Add the end marker */
+ memcpy(buffer, &end_marker, sizeof(u64));
+ }
+
+ return kaddr;
+}
+KBASE_EXPORT_TEST_API(kbase_mmu_dump)
+
+static u64 lock_region(kbase_device *kbdev, u64 pfn, u32 num_pages)
+{
+ u64 region;
+
+ /* can't lock a zero sized range */
+ OSK_ASSERT(num_pages);
+
+ region = pfn << OSK_PAGE_SHIFT;
+ /*
+ * osk_clz returns (given the ASSERT above):
+ * 32-bit: 0 .. 31
+ * 64-bit: 0 .. 63
+ *
+ * 32-bit: 32 + 10 - osk_clz(num_pages)
+ * results in the range (11 .. 42)
+ * 64-bit: 64 + 10 - osk_clz(num_pages)
+ * results in the range (11 .. 42)
+ */
+
+ /* gracefully handle num_pages being zero */
+ if (0 == num_pages)
+ {
+ region |= 11;
+ }
+ else
+ {
+ u8 region_width;
+ region_width = ( OSK_BITS_PER_LONG + 10 - osk_clz(num_pages) );
+ if (num_pages != (1ul << (region_width - 11)))
+ {
+ /* not pow2, so must go up to the next pow2 */
+ region_width += 1;
+ }
+ OSK_ASSERT(region_width <= KBASE_LOCK_REGION_MAX_SIZE);
+ OSK_ASSERT(region_width >= KBASE_LOCK_REGION_MIN_SIZE);
+ region |= region_width;
+ }
+
+ return region;
+}
+
+static void bus_fault_worker(osk_workq_work *data)
+{
+ const int num_as = 16;
+ kbase_as * faulting_as;
+ int as_no;
+ kbase_context * kctx;
+ kbase_device * kbdev;
+ u32 reg;
+ mali_bool reset_status= MALI_FALSE;
+
+ faulting_as = CONTAINER_OF(data, kbase_as, work_busfault);
+ as_no = faulting_as->number;
+
+ kbdev = CONTAINER_OF( faulting_as, kbase_device, as[as_no] );
+
+ /* Grab the context that was already refcounted in kbase_mmu_interrupt().
+ * Therefore, it cannot be scheduled out of this AS until we explicitly release it
+ *
+ * NOTE: NULL can be returned here if we're gracefully handling a spurious interrupt */
+ kctx = kbasep_js_runpool_lookup_ctx_noretain( kbdev, as_no );
+
+ /* switch to UNMAPPED mode, will abort all jobs and stop any hw counter dumping */
+ /* AS transaction begin */
+ osk_mutex_lock(&kbdev->as[as_no].transaction_mutex);
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8245))
+ {
+ /* Due to H/W issue 8245 we need to reset the GPU after using UNMAPPED mode.
+ * We start the reset before switching to UNMAPPED to ensure that unrelated jobs
+ * are evicted from the GPU before the switch.
+ */
+ reset_status = kbase_prepare_to_reset_gpu(kbdev);
+ }
+
+ reg = kbase_reg_read(kbdev, MMU_AS_REG(as_no, ASn_TRANSTAB_LO), kctx);
+ reg &= ~3;
+ kbase_reg_write(kbdev, MMU_AS_REG(as_no, ASn_TRANSTAB_LO), reg, kctx);
+ kbase_reg_write(kbdev, MMU_AS_REG(as_no, ASn_COMMAND), ASn_COMMAND_UPDATE, kctx);
+
+ kbase_reg_write(kbdev, MMU_REG(MMU_IRQ_CLEAR), (1UL << as_no) | (1UL << (as_no + num_as)) , NULL);
+ osk_mutex_unlock(&kbdev->as[as_no].transaction_mutex);
+ /* AS transaction end */
+
+ mmu_mask_reenable( kbdev, kctx, faulting_as );
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8245) && reset_status)
+ {
+ kbase_reset_gpu(kbdev);
+ }
+
+ /* By this point, the fault was handled in some way, so release the ctx refcount */
+ if ( kctx != NULL )
+ {
+ kbasep_js_runpool_release_ctx( kbdev, kctx );
+ }
+}
+
+void kbase_mmu_interrupt(kbase_device * kbdev, u32 irq_stat)
+{
+ const int num_as = 16;
+ kbasep_js_device_data *js_devdata;
+ const int busfault_shift = 16;
+ const int pf_shift = 0;
+ const unsigned long mask = (1UL << num_as) - 1;
+
+ u64 fault_addr;
+ u32 new_mask;
+ u32 tmp;
+
+ u32 bf_bits = (irq_stat >> busfault_shift) & mask; /* bus faults */
+ /* Ignore ASes with both pf and bf */
+ u32 pf_bits = ((irq_stat >> pf_shift) & mask) & ~bf_bits; /* page faults */
+
+ OSK_ASSERT( NULL != kbdev);
+
+ js_devdata = &kbdev->js_data;
+
+ /* remember current mask */
+ osk_spinlock_irq_lock(&kbdev->mmu_mask_change);
+ new_mask = kbase_reg_read(kbdev, MMU_REG(MMU_IRQ_MASK), NULL);
+ /* mask interrupts for now */
+ kbase_reg_write(kbdev, MMU_REG(MMU_IRQ_MASK), 0, NULL);
+ osk_spinlock_irq_unlock(&kbdev->mmu_mask_change);
+
+ while (bf_bits)
+ {
+ kbase_as * as;
+ int as_no;
+ kbase_context * kctx;
+
+ /* the while logic ensures we have a bit set, no need to check for not-found here */
+ as_no = osk_find_first_set_bit(bf_bits);
+
+ /* Refcount the kctx ASAP - it shouldn't disappear anyway, since Bus/Page faults
+ * _should_ only occur whilst jobs are running, and a job causing the Bus/Page fault
+ * shouldn't complete until the MMU is updated */
+ kctx = kbasep_js_runpool_lookup_ctx( kbdev, as_no );
+
+ /* mark as handled */
+ bf_bits &= ~(1UL << as_no);
+
+ /* find faulting address */
+ fault_addr = kbase_reg_read(kbdev, MMU_AS_REG(as_no, ASn_FAULTADDRESS_HI), kctx);
+ fault_addr <<= 32;
+ fault_addr |= kbase_reg_read(kbdev, MMU_AS_REG(as_no, ASn_FAULTADDRESS_LO), kctx);
+
+ if (kctx)
+ {
+ /* hw counters dumping in progress, signal the other thread that it failed */
+ if ((kbdev->hwcnt.kctx == kctx) && (kbdev->hwcnt.state == KBASE_INSTR_STATE_DUMPING))
+ {
+ kbdev->hwcnt.state = KBASE_INSTR_STATE_FAULT;
+ }
+
+ /* Stop the kctx from submitting more jobs and cause it to be scheduled
+ * out/rescheduled when all references to it are released */
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+ kbasep_js_clear_submit_allowed( js_devdata, kctx );
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+
+ OSK_PRINT_WARN(OSK_BASE_MMU, "Bus error in AS%d at 0x%016llx\n", as_no, fault_addr);
+ }
+ else
+ {
+ OSK_PRINT_WARN(OSK_BASE_MMU,
+ "Bus error in AS%d at 0x%016llx with no context present! "
+ "Suprious IRQ or SW Design Error?\n",
+ as_no, fault_addr);
+ }
+
+ as = &kbdev->as[as_no];
+
+ /* remove the queued BFs from the mask */
+ new_mask &= ~(1UL << (as_no + num_as));
+
+ /* We need to switch to UNMAPPED mode - but we do this in a worker so that we can sleep */
+ osk_workq_work_init(&as->work_busfault, bus_fault_worker);
+ osk_workq_submit(&as->pf_wq, &as->work_busfault);
+ }
+
+ /*
+ * pf_bits is non-zero if we have at least one AS with a page fault and no bus fault.
+ * Handle the PFs in our worker thread.
+ */
+ while (pf_bits)
+ {
+ kbase_as * as;
+ /* the while logic ensures we have a bit set, no need to check for not-found here */
+ int as_no = osk_find_first_set_bit(pf_bits);
+ kbase_context * kctx;
+
+ /* Refcount the kctx ASAP - it shouldn't disappear anyway, since Bus/Page faults
+ * _should_ only occur whilst jobs are running, and a job causing the Bus/Page fault
+ * shouldn't complete until the MMU is updated */
+ kctx = kbasep_js_runpool_lookup_ctx( kbdev, as_no );
+
+ /* mark as handled */
+ pf_bits &= ~(1UL << as_no);
+
+ /* find faulting address */
+ fault_addr = kbase_reg_read(kbdev, MMU_AS_REG(as_no, ASn_FAULTADDRESS_HI), kctx);
+ fault_addr <<= 32;
+ fault_addr |= kbase_reg_read(kbdev, MMU_AS_REG(as_no, ASn_FAULTADDRESS_LO), kctx);
+
+ if ( kctx == NULL )
+ {
+ OSK_PRINT_WARN(OSK_BASE_MMU,
+ "Page fault in AS%d at 0x%016llx with no context present! "
+ "Suprious IRQ or SW Design Error?\n",
+ as_no, fault_addr);
+ }
+
+ as = &kbdev->as[as_no];
+
+ /* remove the queued PFs from the mask */
+ new_mask &= ~((1UL << as_no) | (1UL << (as_no + num_as)));
+
+ /* queue work pending for this AS */
+ as->fault_addr = fault_addr;
+
+ osk_workq_work_init(&as->work_pagefault, page_fault_worker);
+ osk_workq_submit(&as->pf_wq, &as->work_pagefault);
+ }
+
+ /* reenable interrupts */
+ osk_spinlock_irq_lock(&kbdev->mmu_mask_change);
+ tmp = kbase_reg_read(kbdev, MMU_REG(MMU_IRQ_MASK), NULL);
+ new_mask |= tmp;
+ kbase_reg_write(kbdev, MMU_REG(MMU_IRQ_MASK), new_mask, NULL);
+ osk_spinlock_irq_unlock(&kbdev->mmu_mask_change);
+}
+KBASE_EXPORT_TEST_API(kbase_mmu_interrupt)
+
+const char *kbase_exception_name(u32 exception_code)
+{
+ const char *e;
+
+ switch(exception_code)
+ {
+ /* Non-Fault Status code */
+ case 0x00: e = "NOT_STARTED/IDLE/OK"; break;
+ case 0x01: e = "DONE"; break;
+ case 0x02: e = "INTERRUPTED"; break;
+ case 0x03: e = "STOPPED"; break;
+ case 0x04: e = "TERMINATED"; break;
+ case 0x08: e = "ACTIVE"; break;
+ /* Job exceptions */
+ case 0x40: e = "JOB_CONFIG_FAULT"; break;
+ case 0x41: e = "JOB_POWER_FAULT"; break;
+ case 0x42: e = "JOB_READ_FAULT"; break;
+ case 0x43: e = "JOB_WRITE_FAULT"; break;
+ case 0x44: e = "JOB_AFFINITY_FAULT"; break;
+ case 0x48: e = "JOB_BUS_FAULT"; break;
+ case 0x50: e = "INSTR_INVALID_PC"; break;
+ case 0x51: e = "INSTR_INVALID_ENC"; break;
+ case 0x52: e = "INSTR_TYPE_MISMATCH"; break;
+ case 0x53: e = "INSTR_OPERAND_FAULT"; break;
+ case 0x54: e = "INSTR_TLS_FAULT"; break;
+ case 0x55: e = "INSTR_BARRIER_FAULT"; break;
+ case 0x56: e = "INSTR_ALIGN_FAULT"; break;
+ case 0x58: e = "DATA_INVALID_FAULT"; break;
+ case 0x59: e = "TILE_RANGE_FAULT"; break;
+ case 0x5A: e = "ADDR_RANGE_FAULT"; break;
+ case 0x60: e = "OUT_OF_MEMORY"; break;
+ /* GPU exceptions */
+ case 0x80: e = "DELAYED_BUS_FAULT"; break;
+ case 0x81: e = "SHAREABILITY_FAULT"; break;
+ /* MMU exceptions */
+ case 0xC0: case 0xC1: case 0xC2: case 0xC3:
+ case 0xC4: case 0xC5: case 0xC6: case 0xC7:
+ e = "TRANSLATION_FAULT"; break;
+ case 0xC8: e = "PERMISSION_FAULT"; break;
+ case 0xD0: case 0xD1: case 0xD2: case 0xD3:
+ case 0xD4: case 0xD5: case 0xD6: case 0xD7:
+ e = "TRANSTAB_BUS_FAULT"; break;
+ case 0xD8: e = "ACCESS_FLAG"; break;
+ default:
+ e = "UNKNOWN"; break;
+ };
+
+ return e;
+}
+
+/**
+ * The caller must ensure it's retained the ctx to prevent it from being scheduled out whilst it's being worked on.
+ */
+static void kbase_mmu_report_fault_and_kill(kbase_context *kctx, kbase_as * as, mali_addr64 fault_addr)
+{
+ u32 fault_status;
+ u32 reg;
+ int exception_type;
+ int access_type;
+ int source_id;
+ int as_no;
+ kbase_device * kbdev;
+ kbasep_js_device_data *js_devdata;
+ mali_bool reset_status = MALI_FALSE;
+#if MALI_DEBUG
+ static const char *access_type_names[] = { "RESERVED", "EXECUTE", "READ", "WRITE" };
+#endif
+
+ OSK_ASSERT(as);
+ OSK_ASSERT(kctx);
+ CSTD_UNUSED(fault_addr);
+
+ as_no = as->number;
+ kbdev = kctx->kbdev;
+ js_devdata = &kbdev->js_data;
+
+ /* ASSERT that the context won't leave the runpool */
+ OSK_ASSERT( kbasep_js_debug_check_ctx_refcount( kbdev, kctx ) > 0 );
+
+ fault_status = kbase_reg_read(kbdev, MMU_AS_REG(as_no, ASn_FAULTSTATUS), kctx);
+
+ /* decode the fault status */
+ exception_type = fault_status & 0xFF;
+ access_type = (fault_status >> 8) & 0x3;
+ source_id = (fault_status >> 16);
+
+ /* terminal fault, print info about the fault */
+ OSK_PRINT_WARN(OSK_BASE_MMU, "Fault in AS%d at VA 0x%016llX", as_no, fault_addr);
+ OSK_PRINT_WARN(OSK_BASE_MMU, "raw fault status 0x%X", fault_status);
+ OSK_PRINT_WARN(OSK_BASE_MMU, "decoded fault status (%s):", (fault_status & (1 << 10) ? "DECODER FAULT" : "SLAVE FAULT"));
+ OSK_PRINT_WARN(OSK_BASE_MMU, "exception type 0x%X: %s", exception_type, kbase_exception_name(exception_type));
+ OSK_PRINT_WARN(OSK_BASE_MMU, "access type 0x%X: %s", access_type, access_type_names[access_type]);
+ OSK_PRINT_WARN(OSK_BASE_MMU, "source id 0x%X", source_id);
+
+ /* hardware counters dump fault handling */
+ if ((kbdev->hwcnt.kctx) &&
+ (kbdev->hwcnt.kctx->as_nr == as_no) &&
+ (kbdev->hwcnt.state == KBASE_INSTR_STATE_DUMPING))
+ {
+ u32 num_core_groups = kbdev->gpu_props.num_core_groups;
+ if ((fault_addr >= kbdev->hwcnt.addr) && (fault_addr < (kbdev->hwcnt.addr + (num_core_groups * 2048))))
+ {
+ kbdev->hwcnt.state = KBASE_INSTR_STATE_FAULT;
+ }
+ }
+
+ /* Stop the kctx from submitting more jobs and cause it to be scheduled
+ * out/rescheduled - this will occur on releasing the context's refcount */
+ osk_spinlock_irq_lock( &js_devdata->runpool_irq.lock );
+ kbasep_js_clear_submit_allowed( js_devdata, kctx );
+ osk_spinlock_irq_unlock( &js_devdata->runpool_irq.lock );
+
+ /* Kill any running jobs from the context. Submit is disallowed, so no more jobs from this
+ * context can appear in the job slots from this point on */
+ kbase_job_kill_jobs_from_context(kctx);
+ /* AS transaction begin */
+ osk_mutex_lock(&as->transaction_mutex);
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8245))
+ {
+ /* Due to H/W issue 8245 we need to reset the GPU after using UNMAPPED mode.
+ * We start the reset before switching to UNMAPPED to ensure that unrelated jobs
+ * are evicted from the GPU before the switch.
+ */
+ reset_status = kbase_prepare_to_reset_gpu(kbdev);
+ }
+
+ /* switch to UNMAPPED mode, will abort all jobs and stop any hw counter dumping */
+ reg = kbase_reg_read(kbdev, MMU_AS_REG(as_no, ASn_TRANSTAB_LO), kctx);
+ reg &= ~3;
+ kbase_reg_write(kbdev, MMU_AS_REG(as_no, ASn_TRANSTAB_LO), reg, kctx);
+ kbase_reg_write(kbdev, MMU_AS_REG(as_no, ASn_COMMAND), ASn_COMMAND_UPDATE, kctx);
+
+ kbase_reg_write(kbdev, MMU_REG(MMU_IRQ_CLEAR), (1UL << as_no), NULL);
+
+ osk_mutex_unlock(&as->transaction_mutex);
+ /* AS transaction end */
+ mmu_mask_reenable(kbdev, kctx, as);
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8245) && reset_status)
+ {
+ kbase_reset_gpu(kbdev);
+ }
+}
+
+void kbasep_as_do_poke(osk_workq_work * work)
+{
+ kbase_as * as;
+ kbase_device * kbdev;
+
+ OSK_ASSERT(work);
+ as = CONTAINER_OF(work, kbase_as, poke_work);
+ kbdev = CONTAINER_OF(as, kbase_device, as[as->number]);
+
+ kbase_pm_context_active(kbdev);
+
+ /* AS transaction begin */
+ osk_mutex_lock(&as->transaction_mutex);
+ /* Force a uTLB invalidate */
+ kbase_reg_write(kbdev, MMU_AS_REG(as->number, ASn_COMMAND), ASn_COMMAND_UNLOCK, NULL);
+ osk_mutex_unlock(&as->transaction_mutex);
+ /* AS transaction end */
+
+ kbase_pm_context_idle(kbdev);
+
+ if (osk_atomic_get(&as->poke_refcount))
+ {
+ osk_error err;
+ /* still someone depending on the UNLOCK, schedule a run */
+ err = osk_timer_modify(&as->poke_timer, 5/*ms*/);
+ if (err != OSK_ERR_NONE)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MMU, "Failed to enable the BASE_HW_ISSUE_8316 workaround");
+ }
+ }
+}
+
+void kbasep_as_poke_timer_callback(void* arg)
+{
+ kbase_as * as;
+ as = (kbase_as*)arg;
+ osk_workq_submit(&as->poke_wq, &as->poke_work);
+}
+
+void kbase_as_poking_timer_retain(kbase_as * as)
+{
+ OSK_ASSERT(as);
+
+ if (1 == osk_atomic_inc(&as->poke_refcount))
+ {
+ /* need to start poking */
+ osk_workq_submit(&as->poke_wq, &as->poke_work);
+
+ }
+}
+
+void kbase_as_poking_timer_release(kbase_as * as)
+{
+ OSK_ASSERT(as);
+ osk_atomic_dec(&as->poke_refcount);
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_pm.c
+ * Base kernel power management APIs
+ */
+
+#include <osk/mali_osk.h>
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_midg_regmap.h>
+
+#include <kbase/src/common/mali_kbase_pm.h>
+
+/* Policy operation structures */
+extern const kbase_pm_policy kbase_pm_always_on_policy_ops;
+extern const kbase_pm_policy kbase_pm_demand_policy_ops;
+
+/** A list of the power policies available in the system */
+static const kbase_pm_policy * const policy_list[] =
+{
+/* It is not possible to modify power management run-time in a model build. Therefore an
+ * instrumented model build must use always on power management policy. */
+#if MALI_NO_MALI || ( MALI_KBASEP_MODEL && MALI_INSTRUMENTATION_LEVEL )
+ &kbase_pm_always_on_policy_ops,
+ &kbase_pm_demand_policy_ops
+#else
+ &kbase_pm_demand_policy_ops,
+ &kbase_pm_always_on_policy_ops
+#endif
+};
+
+/** The number of policies available in the system.
+ * This is derived from the number of functions listed in policy_get_functions.
+ */
+#define POLICY_COUNT (sizeof(policy_list)/sizeof(*policy_list))
+
+void kbase_pm_register_access_enable(kbase_device *kbdev)
+{
+ kbase_pm_callback_conf *callbacks;
+
+ callbacks = (kbase_pm_callback_conf*) kbasep_get_config_value(kbdev, kbdev->config_attributes,
+ KBASE_CONFIG_ATTR_POWER_MANAGEMENT_CALLBACKS);
+
+ if (callbacks)
+ {
+ callbacks->power_on_callback(kbdev);
+ }
+}
+
+void kbase_pm_register_access_disable(kbase_device *kbdev)
+{
+ kbase_pm_callback_conf *callbacks;
+
+ callbacks = (kbase_pm_callback_conf*) kbasep_get_config_value(kbdev, kbdev->config_attributes,
+ KBASE_CONFIG_ATTR_POWER_MANAGEMENT_CALLBACKS);
+
+ if (callbacks)
+ {
+ callbacks->power_off_callback(kbdev);
+ }
+}
+
+mali_error kbase_pm_init(kbase_device *kbdev)
+{
+ mali_error ret = MALI_ERROR_NONE;
+ osk_error osk_err;
+ kbase_pm_callback_conf *callbacks;
+
+ OSK_ASSERT(kbdev != NULL);
+
+ kbdev->pm.gpu_powered = MALI_FALSE;
+
+ callbacks = (kbase_pm_callback_conf*) kbasep_get_config_value(kbdev, kbdev->config_attributes,
+ KBASE_CONFIG_ATTR_POWER_MANAGEMENT_CALLBACKS);
+ if (callbacks)
+ {
+ kbdev->pm.callback_power_on = callbacks->power_on_callback;
+ kbdev->pm.callback_power_off = callbacks->power_off_callback;
+ }
+ else
+ {
+ kbdev->pm.callback_power_on = NULL;
+ kbdev->pm.callback_power_off = NULL;
+ }
+
+ /* Initialise the metrics subsystem */
+ ret = kbasep_pm_metrics_init(kbdev);
+ if (MALI_ERROR_NONE != ret)
+ {
+ return ret;
+ }
+
+ osk_err = osk_waitq_init(&kbdev->pm.power_up_waitqueue);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto power_up_waitq_fail;
+ }
+
+ osk_err = osk_waitq_init(&kbdev->pm.power_down_waitqueue);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto power_down_waitq_fail;
+ }
+
+ osk_err = osk_waitq_init(&kbdev->pm.policy_outstanding_event);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto policy_outstanding_event_waitq_fail;
+ }
+ osk_waitq_set(&kbdev->pm.policy_outstanding_event);
+
+ osk_err = osk_workq_init(&kbdev->pm.workqueue, "kbase_pm", OSK_WORKQ_NON_REENTRANT);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto workq_fail;
+ }
+
+ osk_err = osk_spinlock_irq_init(&kbdev->pm.power_change_lock, OSK_LOCK_ORDER_POWER_MGMT);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto power_change_lock_fail;
+ }
+
+ osk_err = osk_spinlock_irq_init(&kbdev->pm.active_count_lock, OSK_LOCK_ORDER_POWER_MGMT_ACTIVE);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto active_count_lock_fail;
+ }
+
+ osk_err = osk_spinlock_irq_init(&kbdev->pm.gpu_cycle_counter_requests_lock, OSK_LOCK_ORDER_POWER_MGMT_GPU_CYCLE_COUNTER);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto gpu_cycle_counter_requests_lock_fail;
+ }
+
+ osk_err = osk_spinlock_irq_init(&kbdev->pm.gpu_powered_lock, OSK_LOCK_ORDER_POWER_MGMT);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ goto gpu_powered_lock_fail;
+ }
+
+ return MALI_ERROR_NONE;
+
+gpu_powered_lock_fail:
+ osk_spinlock_irq_term(&kbdev->pm.gpu_cycle_counter_requests_lock);
+gpu_cycle_counter_requests_lock_fail:
+ osk_spinlock_irq_term(&kbdev->pm.active_count_lock);
+active_count_lock_fail:
+ osk_spinlock_irq_term(&kbdev->pm.power_change_lock);
+power_change_lock_fail:
+ osk_workq_term(&kbdev->pm.workqueue);
+workq_fail:
+ osk_waitq_term(&kbdev->pm.power_down_waitqueue);
+policy_outstanding_event_waitq_fail:
+ osk_waitq_term(&kbdev->pm.policy_outstanding_event);
+power_down_waitq_fail:
+ osk_waitq_term(&kbdev->pm.power_up_waitqueue);
+power_up_waitq_fail:
+ kbasep_pm_metrics_term(kbdev);
+ return MALI_ERROR_FUNCTION_FAILED;
+}
+KBASE_EXPORT_TEST_API(kbase_pm_init)
+
+mali_error kbase_pm_powerup(kbase_device *kbdev)
+{
+ mali_error ret;
+
+ OSK_ASSERT(kbdev != NULL);
+
+ ret = kbase_pm_init_hw(kbdev);
+ if (ret != MALI_ERROR_NONE)
+ {
+ return ret;
+ }
+
+ kbase_pm_power_transitioning(kbdev);
+
+ kbasep_pm_read_present_cores(kbdev);
+
+ /* Pretend the GPU is active to prevent a power policy turning the GPU cores off */
+ osk_spinlock_irq_lock(&kbdev->pm.active_count_lock);
+ kbdev->pm.active_count = 1;
+ osk_spinlock_irq_unlock(&kbdev->pm.active_count_lock);
+
+ osk_spinlock_irq_lock(&kbdev->pm.gpu_cycle_counter_requests_lock);
+ /* Ensure cycle counter is off */
+ kbdev->pm.gpu_cycle_counter_requests = 0;
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_COMMAND), GPU_COMMAND_CYCLE_COUNT_STOP, NULL);
+ osk_spinlock_irq_unlock(&kbdev->pm.gpu_cycle_counter_requests_lock);
+
+ osk_atomic_set(&kbdev->pm.pending_events, 0);
+
+ osk_atomic_set(&kbdev->pm.work_active,(u32)KBASE_PM_WORK_ACTIVE_STATE_INACTIVE);
+
+ kbdev->pm.new_policy = NULL;
+ kbdev->pm.current_policy = policy_list[0];
+ kbdev->pm.current_policy->init(kbdev);
+
+ kbase_pm_send_event(kbdev, KBASE_PM_EVENT_POLICY_INIT);
+
+ /* Idle the GPU */
+ kbase_pm_context_idle(kbdev);
+
+ return MALI_ERROR_NONE;
+}
+KBASE_EXPORT_TEST_API(kbase_pm_powerup)
+
+void kbase_pm_power_transitioning(kbase_device *kbdev)
+{
+ OSK_ASSERT(kbdev != NULL);
+
+ /* Clear the wait queues that are used to detect successful power up or down */
+ osk_waitq_clear(&kbdev->pm.power_up_waitqueue);
+ osk_waitq_clear(&kbdev->pm.power_down_waitqueue);
+}
+
+KBASE_EXPORT_TEST_API(kbase_pm_power_transitioning)
+
+void kbase_pm_power_up_done(kbase_device *kbdev)
+{
+ OSK_ASSERT(kbdev != NULL);
+
+ osk_waitq_set(&kbdev->pm.power_up_waitqueue);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_power_up_done)
+
+void kbase_pm_reset_done(kbase_device *kbdev)
+{
+ OSK_ASSERT(kbdev != NULL);
+
+ osk_waitq_set(&kbdev->pm.power_up_waitqueue);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_reset_done)
+
+void kbase_pm_power_down_done(kbase_device *kbdev)
+{
+ OSK_ASSERT(kbdev != NULL);
+
+ osk_waitq_set(&kbdev->pm.power_down_waitqueue);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_power_down_done)
+
+static void kbase_pm_wait_for_no_outstanding_events(kbase_device *kbdev)
+{
+ osk_waitq_wait(&kbdev->pm.policy_outstanding_event);
+}
+
+void kbase_pm_context_active(kbase_device *kbdev)
+{
+ int c;
+
+ OSK_ASSERT(kbdev != NULL);
+
+ osk_spinlock_irq_lock(&kbdev->pm.active_count_lock);
+ c = ++kbdev->pm.active_count;
+ osk_spinlock_irq_unlock(&kbdev->pm.active_count_lock);
+
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, PM_CONTEXT_ACTIVE, NULL, NULL, 0u, c );
+
+ if (c == 1)
+ {
+ /* First context active */
+ kbase_pm_send_event(kbdev, KBASE_PM_EVENT_GPU_ACTIVE);
+
+ kbasep_pm_record_gpu_active(kbdev);
+ }
+ /* Synchronise with the power policy to ensure that the event has been noticed */
+ kbase_pm_wait_for_no_outstanding_events(kbdev);
+
+ kbase_pm_wait_for_power_up(kbdev);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_context_active)
+
+void kbase_pm_context_idle(kbase_device *kbdev)
+{
+ int c;
+
+ OSK_ASSERT(kbdev != NULL);
+
+ osk_spinlock_irq_lock(&kbdev->pm.active_count_lock);
+
+ c = --kbdev->pm.active_count;
+
+ KBASE_TRACE_ADD_REFCOUNT( kbdev, PM_CONTEXT_IDLE, NULL, NULL, 0u, c );
+
+ OSK_ASSERT(c >= 0);
+
+ if (c == 0)
+ {
+ /* Last context has gone idle */
+ kbase_pm_send_event(kbdev, KBASE_PM_EVENT_GPU_IDLE);
+
+ kbasep_pm_record_gpu_idle(kbdev);
+ }
+
+ /* We must wait for the above functions to finish (in the case c==0) before releasing the lock otherwise there is
+ * a race with another thread calling kbase_pm_context_active - in this case the IDLE message could be sent
+ * *after* the ACTIVE message causing the policy and metrics systems to become confused
+ */
+ osk_spinlock_irq_unlock(&kbdev->pm.active_count_lock);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_context_idle)
+
+void kbase_pm_halt(kbase_device *kbdev)
+{
+ OSK_ASSERT(kbdev != NULL);
+
+ if (kbdev->pm.current_policy != NULL)
+ {
+ /* Turn the GPU off */
+ kbase_pm_send_event(kbdev, KBASE_PM_EVENT_SYSTEM_SUSPEND);
+ /* Wait for the policy to acknowledge */
+ kbase_pm_wait_for_power_down(kbdev);
+ }
+}
+KBASE_EXPORT_TEST_API(kbase_pm_halt)
+
+void kbase_pm_term(kbase_device *kbdev)
+{
+ OSK_ASSERT(kbdev != NULL);
+ OSK_ASSERT(kbdev->pm.active_count == 0);
+ OSK_ASSERT(kbdev->pm.gpu_cycle_counter_requests == 0);
+ /* Destroy the workqueue - this ensures that all messages have been processed */
+ osk_workq_term(&kbdev->pm.workqueue);
+
+ if (kbdev->pm.current_policy != NULL)
+ {
+ /* Free any resources the policy allocated */
+ kbdev->pm.current_policy->term(kbdev);
+ }
+
+ /* Free the wait queues */
+ osk_waitq_term(&kbdev->pm.power_up_waitqueue);
+ osk_waitq_term(&kbdev->pm.power_down_waitqueue);
+ osk_waitq_term(&kbdev->pm.policy_outstanding_event);
+
+ /* Synchronise with other threads */
+ osk_spinlock_irq_lock(&kbdev->pm.power_change_lock);
+ osk_spinlock_irq_unlock(&kbdev->pm.power_change_lock);
+
+ /* Free the spinlocks */
+ osk_spinlock_irq_term(&kbdev->pm.power_change_lock);
+ osk_spinlock_irq_term(&kbdev->pm.active_count_lock);
+ osk_spinlock_irq_term(&kbdev->pm.gpu_cycle_counter_requests_lock);
+ osk_spinlock_irq_term(&kbdev->pm.gpu_powered_lock);
+
+ /* Shut down the metrics subsystem */
+ kbasep_pm_metrics_term(kbdev);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_term)
+
+void kbase_pm_wait_for_power_up(kbase_device *kbdev)
+{
+ OSK_ASSERT(kbdev != NULL);
+
+ osk_waitq_wait(&kbdev->pm.power_up_waitqueue);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_wait_for_power_up)
+
+void kbase_pm_wait_for_power_down(kbase_device *kbdev)
+{
+ OSK_ASSERT(kbdev != NULL);
+
+ osk_waitq_wait(&kbdev->pm.power_down_waitqueue);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_wait_for_power_down)
+
+int kbase_pm_list_policies(const kbase_pm_policy * const **list)
+{
+ if (!list)
+ return POLICY_COUNT;
+
+ *list = policy_list;
+
+ return POLICY_COUNT;
+}
+KBASE_EXPORT_TEST_API(kbase_pm_list_policies)
+
+const kbase_pm_policy *kbase_pm_get_policy(kbase_device *kbdev)
+{
+ OSK_ASSERT(kbdev != NULL);
+
+ return kbdev->pm.current_policy;
+}
+KBASE_EXPORT_TEST_API(kbase_pm_get_policy)
+
+void kbase_pm_set_policy(kbase_device *kbdev, const kbase_pm_policy *new_policy)
+{
+ OSK_ASSERT(kbdev != NULL);
+ OSK_ASSERT(new_policy != NULL);
+
+ if (kbdev->pm.new_policy) {
+ /* A policy change is already outstanding */
+ return;
+ }
+ /* During a policy change we pretend the GPU is active */
+ kbase_pm_context_active(kbdev);
+
+ kbdev->pm.new_policy = new_policy;
+ kbase_pm_send_event(kbdev, KBASE_PM_EVENT_POLICY_CHANGE);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_set_policy)
+
+void kbase_pm_change_policy(kbase_device *kbdev)
+{
+ OSK_ASSERT(kbdev != NULL);
+
+ kbdev->pm.current_policy->term(kbdev);
+ kbdev->pm.current_policy = kbdev->pm.new_policy;
+ kbdev->pm.current_policy->init(kbdev);
+ kbase_pm_send_event(kbdev, KBASE_PM_EVENT_POLICY_INIT);
+
+ /* Now the policy change is finished, we release our fake context active reference */
+ kbase_pm_context_idle(kbdev);
+
+ kbdev->pm.new_policy = NULL;
+}
+KBASE_EXPORT_TEST_API(kbase_pm_change_policy)
+
+/** Callback for the power management work queue.
+ *
+ * This function is called on the power management work queue and is responsible for delivering events to the active
+ * power policy. It manipulates the @ref kbase_pm_device_data.work_active field of @ref kbase_pm_device_data to track
+ * whether all events have been consumed.
+ *
+ * @param data A pointer to the @c pm.work field of the @ref kbase_device struct
+ */
+
+STATIC void kbase_pm_worker(osk_workq_work *data)
+{
+ kbase_device *kbdev = CONTAINER_OF(data, kbase_device, pm.work);
+ int pending_events;
+ int old_value;
+ int i;
+
+ do
+ {
+ osk_atomic_set(&kbdev->pm.work_active, (u32)KBASE_PM_WORK_ACTIVE_STATE_PROCESSING);
+
+ /* Atomically read and clear the bit mask */
+ pending_events = osk_atomic_get(&kbdev->pm.pending_events);
+
+ do
+ {
+ old_value = pending_events;
+ pending_events = osk_atomic_compare_and_swap(&kbdev->pm.pending_events, old_value, 0);
+ } while (old_value != pending_events);
+
+ for(i = 0; pending_events; i++)
+ {
+ if (pending_events & (1 << i))
+ {
+ kbdev->pm.current_policy->event(kbdev, (kbase_pm_event)i);
+
+ pending_events &= ~(1 << i);
+ }
+ }
+ i = osk_atomic_compare_and_swap(&kbdev->pm.work_active,
+ (u32)KBASE_PM_WORK_ACTIVE_STATE_PROCESSING,
+ (u32)KBASE_PM_WORK_ACTIVE_STATE_INACTIVE);
+ } while (i == (u32)KBASE_PM_WORK_ACTIVE_STATE_PENDING_EVT);
+ osk_waitq_set(&kbdev->pm.policy_outstanding_event);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_worker)
+
+/** Merge an event into the list of events to deliver.
+ *
+ * This ensures that if, for example, a GPU_IDLE is immediately followed by a GPU_ACTIVE then instead of delivering
+ * both messages to the policy the GPU_IDLE is simply discarded.
+ *
+ * In particular in the sequence GPU_IDLE, GPU_ACTIVE, GPU_IDLE the resultant message is GPU_IDLE and not (GPU_IDLE
+ * and GPU_ACTIVE).
+ *
+ * @param old_events The bit mask of events that were previously pending
+ * @param new_event The event that should be merged into old_events
+ *
+ * @return The combination of old_events and the new event
+ */
+STATIC int kbasep_pm_merge_event(int old_events, kbase_pm_event new_event)
+{
+ switch(new_event) {
+ case KBASE_PM_EVENT_POLICY_INIT:
+ /* On policy initialisation, ignore any pending old_events. */
+ return ( 1 << KBASE_PM_EVENT_POLICY_INIT);
+
+ case KBASE_PM_EVENT_GPU_STATE_CHANGED:
+ case KBASE_PM_EVENT_POLICY_CHANGE:
+ case KBASE_PM_EVENT_CHANGE_GPU_STATE:
+ /* Just merge these events into the list */
+ return old_events | (1 << new_event);
+ case KBASE_PM_EVENT_SYSTEM_SUSPEND:
+ if (old_events & (1 << KBASE_PM_EVENT_SYSTEM_RESUME))
+ {
+ return old_events & ~(1 << KBASE_PM_EVENT_SYSTEM_RESUME);
+ }
+ return old_events | (1 << new_event);
+ case KBASE_PM_EVENT_SYSTEM_RESUME:
+ if (old_events & (1 << KBASE_PM_EVENT_SYSTEM_SUSPEND))
+ {
+ return old_events & ~(1 << KBASE_PM_EVENT_SYSTEM_SUSPEND);
+ }
+ return old_events | (1 << new_event);
+ case KBASE_PM_EVENT_GPU_ACTIVE:
+ if (old_events & (1 << KBASE_PM_EVENT_GPU_IDLE))
+ {
+ return old_events & ~(1 << KBASE_PM_EVENT_GPU_IDLE);
+ }
+ return old_events | (1 << new_event);
+ case KBASE_PM_EVENT_GPU_IDLE:
+ if (old_events & (1 << KBASE_PM_EVENT_GPU_ACTIVE))
+ {
+ return old_events & ~(1 << KBASE_PM_EVENT_GPU_ACTIVE);
+ }
+ return old_events | (1 << new_event);
+ default:
+ /* Unrecognised event - this should never happen */
+ OSK_ASSERT(0);
+ return old_events | (1 << new_event);
+ }
+}
+KBASE_EXPORT_TEST_API(kbasep_pm_merge_event)
+
+void kbase_pm_send_event(kbase_device *kbdev, kbase_pm_event event)
+{
+ int pending_events;
+ int work_active;
+ int old_value, new_value;
+
+ OSK_ASSERT(kbdev != NULL);
+
+ pending_events = osk_atomic_get(&kbdev->pm.pending_events);
+
+ /* Atomically OR the new event into the pending_events bit mask */
+ do
+ {
+ old_value = pending_events;
+ new_value = kbasep_pm_merge_event(pending_events, event);
+ if (old_value == new_value)
+ {
+ /* Event already pending */
+ return;
+ }
+ pending_events = osk_atomic_compare_and_swap(&kbdev->pm.pending_events, old_value, new_value);
+ } while (old_value != pending_events);
+
+ work_active = osk_atomic_get(&kbdev->pm.work_active);
+ do
+ {
+ old_value = work_active;
+ switch(old_value)
+ {
+ case KBASE_PM_WORK_ACTIVE_STATE_INACTIVE:
+ /* Need to enqueue an event */
+ new_value = KBASE_PM_WORK_ACTIVE_STATE_ENQUEUED;
+ break;
+ case KBASE_PM_WORK_ACTIVE_STATE_ENQUEUED:
+ /* Event already queued */
+ return;
+ case KBASE_PM_WORK_ACTIVE_STATE_PROCESSING:
+ /* Event being processed, we need to ensure it checks for another event */
+ new_value = KBASE_PM_WORK_ACTIVE_STATE_PENDING_EVT;
+ break;
+ case KBASE_PM_WORK_ACTIVE_STATE_PENDING_EVT:
+ /* Event being processed, but another check for events is going to happen */
+ return;
+ default:
+ OSK_ASSERT(0);
+ }
+ work_active = osk_atomic_compare_and_swap(&kbdev->pm.work_active, old_value, new_value);
+ } while (old_value != work_active);
+
+ if (old_value == KBASE_PM_WORK_ACTIVE_STATE_INACTIVE)
+ {
+ osk_waitq_clear(&kbdev->pm.policy_outstanding_event);
+ osk_workq_work_init(&kbdev->pm.work, kbase_pm_worker);
+ osk_workq_submit(&kbdev->pm.workqueue, &kbdev->pm.work);
+ }
+}
+
+KBASE_EXPORT_TEST_API(kbase_pm_send_event)
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_pm.h
+ * Power management API definitions
+ */
+
+#ifndef _KBASE_PM_H_
+#define _KBASE_PM_H_
+
+#include <kbase/src/common/mali_midg_regmap.h>
+
+#include "mali_kbase_pm_always_on.h"
+#include "mali_kbase_pm_demand.h"
+
+/* Forward definition - see mali_kbase.h */
+struct kbase_device;
+
+/** The types of core in a GPU.
+ *
+ * These enumerated values are used in calls to @ref kbase_pm_invoke_power_up, @ref kbase_pm_invoke_power_down, @ref
+ * kbase_pm_get_present_cores, @ref kbase_pm_get_active_cores, @ref kbase_pm_get_trans_cores, @ref
+ * kbase_pm_get_ready_cores. The specify which type of core should be acted on.
+ * These values are set in a manner that allows @ref core_type_to_reg function to be simpler and more efficient.
+ */
+typedef enum kbase_pm_core_type
+{
+ KBASE_PM_CORE_L3 = L3_PRESENT_LO, /**< The L3 cache */
+ KBASE_PM_CORE_L2 = L2_PRESENT_LO, /**< The L2 cache */
+ KBASE_PM_CORE_SHADER = SHADER_PRESENT_LO, /**< Shader cores */
+ KBASE_PM_CORE_TILER = TILER_PRESENT_LO /**< Tiler cores */
+} kbase_pm_core_type;
+
+/** Initialize the power management framework.
+ *
+ * Must be called before any other power management function
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ *
+ * @return MALI_ERROR_NONE if the power management framework was successfully initialized.
+ */
+mali_error kbase_pm_init(struct kbase_device *kbdev);
+
+/** Power up GPU after all modules have been initialized and interrupt handlers installed.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ *
+ * @return MALI_ERROR_NONE if powerup was successful.
+ */
+mali_error kbase_pm_powerup(struct kbase_device *kbdev);
+
+/**
+ * Halt the power management framework.
+ * Should ensure that no new interrupts are generated,
+ * but allow any currently running interrupt handlers to complete successfully.
+ * No event can make the pm system turn on the GPU after this function returns.
+ * The active policy is sent @ref KBASE_PM_EVENT_SYSTEM_SUSPEND.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_halt(struct kbase_device *kbdev);
+
+/** Terminate the power management framework.
+ *
+ * No power management functions may be called after this
+ * (except @ref kbase_pm_init)
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_term(struct kbase_device *kbdev);
+
+/** Events that can be sent to a power policy.
+ *
+ * Power policies are expected to handle all these events, although they may choose to take no action.
+ */
+typedef enum kbase_pm_event
+{
+ /* helper for tests */
+ KBASEP_PM_EVENT_FIRST,
+
+ /** Initialize the power policy.
+ *
+ * This event is sent immediately after the @ref kbase_pm_policy.init function of the policy returns.
+ *
+ * The policy may decide to transition the cores to its 'normal' state (e.g. an always on policy would turn all
+ * the cores on). The policy should assume that the GPU is in active use (i.e. as if the @ref
+ * KBASE_PM_EVENT_GPU_ACTIVE event had been received), if this is not the case then @ref KBASE_PM_EVENT_GPU_IDLE
+ * will be called after this event has been handled.
+ */
+ KBASE_PM_EVENT_POLICY_INIT = KBASEP_PM_EVENT_FIRST,
+ /** The power state of the device has changed.
+ *
+ * This event is sent when the GPU raises an interrupt to announce that a power transition has finished. Because
+ * there may be multiple power transitions the power policy must interrogate the state of the GPU to check whether
+ * all expected transitions have finished. If the GPU has just turned on or off then the policy must call @ref
+ * kbase_pm_power_up_done or @ref kbase_pm_power_down_done as appropriate.
+ */
+ KBASE_PM_EVENT_GPU_STATE_CHANGED,
+ /** The GPU is becoming active.
+ *
+ * This event is sent when the first context is about to use the GPU.
+ *
+ * If the core is turned off then this event must cause the core to turn on. This is done asynchronously and the
+ * policy must call the function kbase_pm_power_up_done to signal that the core is turned on sufficiently to allow
+ * register access.
+ */
+ KBASE_PM_EVENT_GPU_ACTIVE,
+ /** The GPU is becoming idle.
+ *
+ * This event is sent when the last context has finished using the GPU.
+ *
+ * The power policy may turn the GPU off entirely (e.g. turn the clocks or power off).
+ */
+ KBASE_PM_EVENT_GPU_IDLE,
+ /** The system has requested a change of power policy.
+ *
+ * The current policy receives this message when a request to change policy occurs. It must ensure that all active
+ * power transitions are completed and then call the @ref kbase_pm_change_policy function.
+ *
+ * This event is only delivered when the policy has been informed that the GPU is 'active' (the power management
+ * code internally increments the context active counter during a policy change).
+ */
+ KBASE_PM_EVENT_POLICY_CHANGE,
+ /** The system is requesting to suspend the GPU.
+ *
+ * The power policy should ensure that the GPU is shut down sufficiently for the system to suspend the device.
+ * Once the GPU is ready the policy should call @ref kbase_pm_power_down_done.
+ */
+ KBASE_PM_EVENT_SYSTEM_SUSPEND,
+ /** The system is requesting to resume the GPU.
+ *
+ * The power policy should restore the GPU to the state it was before the previous
+ * @ref KBASE_PM_EVENT_SYSTEM_SUSPEND event. If the GPU is being powered up then it should call
+ * @ref kbase_pm_power_transitioning before changing the state and @ref kbase_pm_power_up_done when
+ * the transition is complete.
+ */
+ KBASE_PM_EVENT_SYSTEM_RESUME,
+ /** The job scheduler is requesting to power up/down cores.
+ *
+ * This event is sent when:
+ * - powered down cores are needed to complete a job
+ * - powered up cores are not needed anymore
+ */
+ KBASE_PM_EVENT_CHANGE_GPU_STATE,
+
+ /* helpers for tests */
+ KBASEP_PM_EVENT_LAST = KBASE_PM_EVENT_CHANGE_GPU_STATE,
+ KBASEP_PM_EVENT_INVALID
+} kbase_pm_event;
+
+typedef union kbase_pm_policy_data
+{
+ kbasep_pm_policy_always_on always_on;
+ kbasep_pm_policy_demand demand;
+} kbase_pm_policy_data;
+
+/** Power policy structure.
+ *
+ * Each power management policy exposes a (static) instance of this structure which contains function pointers to the
+ * policy's methods.
+ */
+typedef struct kbase_pm_policy
+{
+ /** The name of this policy */
+ char *name;
+
+ /** Function called when the policy is selected
+ *
+ * This should initialize the kbdev->pm.policy_data pointer to the policy's data structure. It should not attempt
+ * to make any changes to hardware state.
+ *
+ * It is undefined what state the cores are in when the function is called, however no power transitions should be
+ * occurring.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+ void (*init)(struct kbase_device *kbdev);
+ /** Function called when the policy is unselected.
+ *
+ * This should free any data allocated with \c init
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+ void (*term)(struct kbase_device *kbdev);
+ /** Function called when there is an event to process
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ * @param event The event to process
+ */
+ void (*event)(struct kbase_device *kbdev, kbase_pm_event event);
+} kbase_pm_policy;
+
+/** Metrics data collected for use by the power management framework.
+ *
+ */
+typedef struct kbasep_pm_metrics_data
+{
+ int vsync_hit;
+ int utilisation;
+
+ osk_ticks time_period_start;
+ u32 time_busy;
+ u32 time_idle;
+ mali_bool gpu_active;
+
+ osk_spinlock_irq lock;
+
+ osk_timer timer;
+ mali_bool timer_active;
+
+ void * platform_data;
+} kbasep_pm_metrics_data;
+
+/** Actions for DVFS.
+ *
+ * kbase_pm_get_dvfs_action will return one of these enumerated values to
+ * describe the action that the DVFS system should take.
+ */
+typedef enum kbase_pm_dvfs_action
+{
+ KBASE_PM_DVFS_NOP, /**< No change in clock frequency is requested */
+ KBASE_PM_DVFS_CLOCK_UP, /**< The clock frequency should be increased if possible */
+ KBASE_PM_DVFS_CLOCK_DOWN /**< The clock frequency should be decreased if possible */
+} kbase_pm_dvfs_action;
+
+/** A value for an atomic @ref work_active,
+ * which tracks whether the work unit has been enqueued.
+ */
+typedef enum kbase_pm_work_active_state
+{
+ KBASE_PM_WORK_ACTIVE_STATE_INACTIVE = 0x00u, /**< There are no work units enqueued and @ref kbase_pm_worker is not running. */
+ KBASE_PM_WORK_ACTIVE_STATE_ENQUEUED = 0x01u, /**< There is a work unit enqueued, but @ref kbase_pm_worker is not running. */
+ KBASE_PM_WORK_ACTIVE_STATE_PROCESSING = 0x02u, /**< @ref kbase_pm_worker is running. */
+ KBASE_PM_WORK_ACTIVE_STATE_PENDING_EVT = 0x03u /**< Processing and there's an event outstanding.
+ @ref kbase_pm_worker is running, but @ref pending_events
+ has been updated since it started so
+ it should recheck the list of pending events before exiting. */
+} kbase_pm_work_active_state;
+
+/** Data stored per device for power management.
+ *
+ * This structure contains data for the power management framework. There is one instance of this structure per device
+ * in the system.
+ */
+typedef struct kbase_pm_device_data
+{
+ /** The policy that is currently actively controlling the power state. */
+ const kbase_pm_policy *current_policy;
+ /** The policy that the system is transitioning to. */
+ const kbase_pm_policy *new_policy;
+ /** The data needed for the current policy. This is considered private to the policy. */
+ kbase_pm_policy_data policy_data;
+ /** The workqueue that the policy callbacks are executed on. */
+ osk_workq workqueue;
+ /** A bit mask of events that are waiting to be delivered to the active policy. */
+ osk_atomic pending_events;
+ /** The work unit that is enqueued onto the workqueue. */
+ osk_workq_work work;
+ /** An atomic which tracks whether the work unit has been enqueued.
+ * For list of possible values please refer to @ref kbase_pm_work_active_state.
+ */
+ osk_atomic work_active;
+ /** The wait queue for power up events. */
+ osk_waitq power_up_waitqueue;
+ /** The wait queue for power down events. */
+ osk_waitq power_down_waitqueue;
+ /** Wait queue for whether there is an outstanding event for the policy */
+ osk_waitq policy_outstanding_event;
+
+ /** The reference count of active contexts on this device. */
+ int active_count;
+ /** Lock to protect active_count */
+ osk_spinlock_irq active_count_lock;
+ /** The reference count of active gpu cycle counter users */
+ int gpu_cycle_counter_requests;
+ /** Lock to protect gpu_cycle_counter_requests */
+ osk_spinlock_irq gpu_cycle_counter_requests_lock;
+ /** A bit mask identifying the shader cores that the power policy would like to be on.
+ * The current state of the cores may be different, but there should be transitions in progress that will
+ * eventually achieve this state (assuming that the policy doesn't change its mind in the mean time.
+ */
+ u64 desired_shader_state;
+ /** A bit mask identifying the tiler cores that the power policy would like to be on.
+ * @see kbase_pm_device_data:desired_shader_state */
+ u64 desired_tiler_state;
+
+ /** Lock protecting the power state of the device.
+ *
+ * This lock must be held when accessing the shader_available_bitmap, tiler_available_bitmap, shader_inuse_bitmap
+ * and tiler_inuse_bitmap fields of kbase_device. It is also held when the hardware power registers are being
+ * written to, to ensure that two threads do not conflict over the power transitions that the hardware should
+ * make.
+ */
+ osk_spinlock_irq power_change_lock;
+
+ /** Set to true when the GPU is powered and register accesses are possible, false otherwise */
+ mali_bool gpu_powered;
+ /** Spinlock that must be held when writing gpu_powered */
+ osk_spinlock_irq gpu_powered_lock;
+
+ /** Structure to hold metrics for the GPU */
+ kbasep_pm_metrics_data metrics;
+
+ /** Callback when the GPU needs to be turned on. See @ref kbase_pm_callback_conf
+ *
+ * @param kbdev The kbase device
+ *
+ * @return 1 if GPU state was lost, 0 otherwise
+ */
+ int (*callback_power_on)(struct kbase_device *kbdev);
+
+ /** Callback when the GPU may be turned off. See @ref kbase_pm_callback_conf
+ *
+ * @param kbdev The kbase device
+ */
+ void (*callback_power_off)(struct kbase_device *kbdev);
+} kbase_pm_device_data;
+
+/** Get the current policy.
+ * Returns the policy that is currently active.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ *
+ * @return The current policy
+ */
+const kbase_pm_policy *kbase_pm_get_policy(struct kbase_device *kbdev);
+
+/** Change the policy to the one specified.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ * @param policy The policy to change to (valid pointer returned from @ref kbase_pm_list_policies)
+ */
+void kbase_pm_set_policy(struct kbase_device *kbdev, const kbase_pm_policy *policy);
+
+/** Retrieve a static list of the available policies.
+ * @param[out] policies An array pointer to take the list of policies. This may be NULL.
+ * The contents of this array must not be modified.
+ *
+ * @return The number of policies
+ */
+int kbase_pm_list_policies(const kbase_pm_policy * const **policies);
+
+/** The current policy is ready to change to the new policy
+ *
+ * The current policy must ensure that all cores have finished transitioning before calling this function.
+ * The new policy is sent an @ref KBASE_PM_EVENT_POLICY_INIT event.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_change_policy(struct kbase_device *kbdev);
+
+/** The GPU is idle.
+ *
+ * The OS may choose to turn off idle devices
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_dev_idle(struct kbase_device *kbdev);
+
+/** The GPU is active.
+ *
+ * The OS should avoid opportunistically turning off the GPU while it is active
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_dev_activate(struct kbase_device *kbdev);
+
+/** Send an event to the active power policy.
+ *
+ * The event is queued for sending to the active power policy. The event is merged with the current queue by the @ref
+ * kbasep_pm_merge_event function which may decide to drop events.
+ *
+ * Note that this function may be called in an atomic context on Linux which implies that it must not sleep.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ * @param event The event that should be queued
+ */
+void kbase_pm_send_event(struct kbase_device *kbdev, kbase_pm_event event);
+
+/** Turn one or more cores on.
+ *
+ * This function is called by the active power policy to turn one or more cores on.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ * @param type The type of core (see the @ref kbase_pm_core_type enumeration)
+ * @param cores A bitmask of cores to turn on
+ */
+void kbase_pm_invoke_power_up(struct kbase_device *kbdev, kbase_pm_core_type type, u64 cores);
+
+/** Turn one or more cores off.
+ *
+ * This function is called by the active power policy to turn one or more core off.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ * @param type The type of core (see the @ref kbase_pm_core_type enumeration)
+ * @param cores A bitmask of cores to turn off
+ */
+void kbase_pm_invoke_power_down(struct kbase_device *kbdev, kbase_pm_core_type type, u64 cores);
+
+/** Get details of the cores that are present in the device.
+ *
+ * This function can be called by the active power policy to return a bitmask of the cores (of a specified type)
+ * present in the GPU device and also a count of the number of cores.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ * @param type The type of core (see the @ref kbase_pm_core_type enumeration)
+ *
+ * @return The bit mask of cores present
+ */
+u64 kbase_pm_get_present_cores(struct kbase_device *kbdev, kbase_pm_core_type type);
+
+/** Get details of the cores that are currently active in the device.
+ *
+ * This function can be called by the active power policy to return a bitmask of the cores (of a specified type) that
+ * are actively processing work (i.e. turned on *and* busy).
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ * @param type The type of core (see the @ref kbase_pm_core_type enumeration)
+ *
+ * @return The bit mask of active cores
+ */
+u64 kbase_pm_get_active_cores(struct kbase_device *kbdev, kbase_pm_core_type type);
+
+/** Get details of the cores that are currently transitioning between power states.
+ *
+ * This function can be called by the active power policy to return a bitmask of the cores (of a specified type) that
+ * are currently transitioning between power states.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ * @param type The type of core (see the @ref kbase_pm_core_type enumeration)
+ *
+ * @return The bit mask of transitioning cores
+ */
+u64 kbase_pm_get_trans_cores(struct kbase_device *kbdev, kbase_pm_core_type type);
+
+/** Get details of the cores that are currently powered and ready for jobs.
+ *
+ * This function can be called by the active power policy to return a bitmask of the cores (of a specified type) that
+ * are powered and ready for jobs (they may or may not be currently executing jobs).
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ * @param type The type of core (see the @ref kbase_pm_core_type enumeration)
+ *
+ * @return The bit mask of ready cores
+ */
+u64 kbase_pm_get_ready_cores(struct kbase_device *kbdev, kbase_pm_core_type type);
+
+/** Return whether the power manager is active
+ *
+ * This function will return true when there are cores (of any time) that are currently transitioning between power
+ * states.
+ *
+ * It can be used on receipt of the @ref KBASE_PM_EVENT_GPU_STATE_CHANGED message to determine whether the requested power
+ * transitions have completely finished or not.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ *
+ * @return true when there are cores transitioning between power states, false otherwise
+ */
+mali_bool kbase_pm_get_pwr_active(struct kbase_device *kbdev);
+
+/** Turn the clock for the device on.
+ *
+ * This function can be used by a power policy to turn the clock for the GPU on. It should be modified during
+ * integration to perform the necessary actions to ensure that the GPU is fully powered and clocked.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_clock_on(struct kbase_device *kbdev);
+
+/** Turn the clock for the device off.
+ *
+ * This function can be used by a power policy to turn the clock for the GPU off. It should be modified during
+ * integration to perform the necessary actions to turn the clock off (if this is possible in the integration).
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_clock_off(struct kbase_device *kbdev);
+
+/** Enable interrupts on the device.
+ *
+ * This function should be called by the active power policy immediately after calling @ref kbase_pm_clock_on to
+ * ensure that interrupts are enabled on the device.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_enable_interrupts(struct kbase_device *kbdev);
+
+/** Disable interrupts on the device.
+ *
+ * This function should be called by the active power policy after shutting down the device (i.e. in the @ref
+ * KBASE_PM_EVENT_GPU_STATE_CHANGED handler after confirming that all cores have powered off). It prevents interrupt
+ * delivery to the CPU so no further @ref KBASE_PM_EVENT_GPU_STATE_CHANGED messages will be received until @ref
+ * kbase_pm_enable_interrupts is called.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_disable_interrupts(struct kbase_device *kbdev);
+
+/** Initialize the hardware
+ *
+ * This function checks the GPU ID register to ensure that the GPU is supported by the driver and performs a reset on
+ * the device so that it is in a known state before the device is used.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ *
+ * @return MALI_ERROR_NONE if the device is supported and successfully reset.
+ */
+mali_error kbase_pm_init_hw(struct kbase_device *kbdev);
+
+/** Inform the power management system that the power state of the device is transitioning.
+ *
+ * This function must be called by the active power policy before transitioning the core between an 'off state' and an
+ * 'on state'. It resets the wait queues that are waited on by @ref kbase_pm_wait_for_power_up and @ref
+ * kbase_pm_wait_for_power_down.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_power_transitioning(struct kbase_device *kbdev);
+
+/** The GPU has been powered up successfully.
+ *
+ * This function must be called by the active power policy when the GPU has been powered up successfully. It signals
+ * to the rest of the system that jobs can start being submitted to the device.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_power_up_done(struct kbase_device *kbdev);
+
+/** The GPU has been reset successfully.
+ *
+ * This function must be called by the GPU interrupt handler when the RESET_COMPLETED bit is set. It signals to the
+ * power management initialization code that the GPU has been successfully reset.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_reset_done(struct kbase_device *kbdev);
+
+/** The GPU has been powered down successfully.
+ *
+ * This function must be called by the active power policy when the GPU has been powered down successfully. It signals
+ * to the rest of the system that a system suspend can now take place.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_power_down_done(struct kbase_device *kbdev);
+
+/** Wait for the power policy to signal power up.
+ *
+ * This function waits for the power policy to signal power up by calling @ref kbase_pm_power_up_done. After the power
+ * policy has signalled this the function will return immediately until the power policy calls @ref
+ * kbase_pm_power_transitioning.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_wait_for_power_up(struct kbase_device *kbdev);
+
+/** Wait for the power policy to signal power down.
+ *
+ * This function waits for the power policy to signal power down by calling @ref kbase_pm_power_down_done. After the
+ * power policy has signalled this the function will return immediately until the power policy calls @ref
+ * kbase_pm_power_transitioning.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_wait_for_power_down(struct kbase_device *kbdev);
+
+/** Increment the count of active contexts.
+ *
+ * This function should be called when a context is about to submit a job. It informs the active power policy that the
+ * GPU is going to be in use shortly and the policy is expected to start turning on the GPU.
+ *
+ * This function will block until the GPU is available.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_context_active(struct kbase_device *kbdev);
+
+/** Decrement the reference count of active contexts.
+ *
+ * This function should be called when a context becomes idle. After this call the GPU may be turned off by the power
+ * policy so the calling code should ensure that it does not access the GPU's registers.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_context_idle(struct kbase_device *kbdev);
+
+/** Check if there are any power transitions to make, and if so start them.
+ *
+ * This function will check the desired_xx_state members of kbase_pm_device_data and the actual status of the
+ * hardware to see if any power transitions can be made at this time to make the hardware state closer to the state
+ * desired by the power policy.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_check_transitions(struct kbase_device *kbdev);
+
+/** Read the bitmasks of present cores.
+ *
+ * This information is cached to avoid having to perform register reads whenever the information is required.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbasep_pm_read_present_cores(struct kbase_device *kbdev);
+
+/** Mark one or more cores as being required for jobs to be submitted.
+ *
+ * This function is called by the job scheduler to mark one or both cores
+ * as being required to submit jobs that are ready to run.
+ *
+ * The cores requested are reference counted and a subsequent call to @ref kbase_pm_register_inuse_cores or
+ * @ref kbase_pm_unrequest_cores should be made to dereference the cores as being 'needed'.
+ *
+ * The current running policy is sent an @ref KBASE_PM_EVENT_CHANGE_GPU_STATE if power up of requested core is
+ * required.
+
+ * The policy is expected to make these cores available at some point in the future,
+ * but may take an arbitrary length of time to reach this state.
+ *
+ * @param kbdev The kbase device structure for the device
+ * @param shader_cores A bitmask of shader cores which are necessary for the job
+ * @param tiler_cores A bitmask of tiler cores which are necessary for the job
+ *
+ * @return MALI_ERROR_NONE if the cores were successfully requested.
+ */
+mali_error kbase_pm_request_cores(struct kbase_device *kbdev, u64 shader_cores, u64 tiler_cores);
+
+/** Unmark one or more cores as being required for jobs to be submitted.
+ *
+ * This function undoes the effect of @ref kbase_pm_request_cores. It should be used when a job is not
+ * going to be submitted to the hardware (e.g. the job is cancelled before it is enqueued).
+ *
+ * The current running policy is sent an @ref KBASE_PM_EVENT_CHANGE_GPU_STATE if power down of requested core
+ * is required.
+ *
+ * The policy may use this as an indication that it can power down cores.
+ *
+ * @param kbdev The kbase device structure for the device
+ * @param shader_cores A bitmask of shader cores (as given to @ref kbase_pm_request_cores)
+ * @param tiler_cores A bitmask of tiler cores (as given to @ref kbase_pm_request_cores)
+ */
+void kbase_pm_unrequest_cores(struct kbase_device *kbdev, u64 shader_cores, u64 tiler_cores);
+
+/** Register a set of cores as in use by a job.
+ *
+ * This function should be called after @ref kbase_pm_request_cores when the job is about to be submitted to
+ * the hardware. It will check that the necessary cores are available and if so update the 'needed' and 'inuse'
+ * bitmasks to reflect that the job is now committed to being run.
+ *
+ * If the necessary cores are not currently available then the function will return MALI_FALSE and have no effect.
+ *
+ * @param kbdev The kbase device structure for the device
+ * @param shader_cores A bitmask of shader cores (as given to @ref kbase_pm_request_cores)
+ * @param tiler_cores A bitmask of tiler cores (as given to @ref kbase_pm_request_cores)
+ *
+ * @return MALI_TRUE if the job can be submitted to the hardware or MALI_FALSE if the job is not ready to run.
+ */
+mali_bool kbase_pm_register_inuse_cores(struct kbase_device *kbdev, u64 shader_cores, u64 tiler_cores);
+
+/** Release cores after a job has run.
+ *
+ * This function should be called when a job has finished running on the hardware. A call to @ref
+ * kbase_pm_register_inuse_cores must have previously occurred. The reference counts of the specified cores will be
+ * decremented which may cause the bitmask of 'inuse' cores to be reduced. The power policy may then turn off any
+ * cores which are no longer 'inuse'.
+ *
+ * @param kbdev The kbase device structure for the device
+ * @param shader_cores A bitmask of shader cores (as given to @ref kbase_pm_register_inuse_cores)
+ * @param tiler_cores A bitmask of tiler cores (as given to @ref kbase_pm_register_inuse_cores)
+ */
+void kbase_pm_release_cores(struct kbase_device *kbdev, u64 shader_cores, u64 tiler_cores);
+
+/** Initialize the metrics gathering framework.
+ *
+ * This must be called before other metric gathering APIs are called.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ *
+ * @return MALI_ERROR_NONE on success, MALI_ERROR_FUNCTION_FAILED on error
+ */
+mali_error kbasep_pm_metrics_init(struct kbase_device *kbdev);
+
+/** Terminate the metrics gathering framework.
+ *
+ * This must be called when metric gathering is no longer required. It is an error to call any metrics gathering
+ * function (other than kbasep_pm_metrics_init) after calling this function.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbasep_pm_metrics_term(struct kbase_device *kbdev);
+
+/** Record that the GPU is active.
+ *
+ * This records that the GPU is now active. The previous GPU state must have been idle, the function will assert if
+ * this is not true in a debug build.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbasep_pm_record_gpu_active(struct kbase_device *kbdev);
+
+/** Record that the GPU is idle.
+ *
+ * This records that the GPU is now idle. The previous GPU state must have been active, the function will assert if
+ * this is not true in a debug build.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbasep_pm_record_gpu_idle(struct kbase_device *kbdev);
+
+/** Function to be called by the frame buffer driver to update the vsync metric.
+ *
+ * This function should be called by the frame buffer driver to update whether the system is hitting the vsync target
+ * or not. buffer_updated should be true if the vsync corresponded with a new frame being displayed, otherwise it
+ * should be false. This function does not need to be called every vsync, but only when the value of buffer_updated
+ * differs from a previous call.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ * @param buffer_updated True if the buffer has been updated on this VSync, false otherwise
+ */
+void kbase_pm_report_vsync(struct kbase_device *kbdev, int buffer_updated);
+
+/** Configure the frame buffer device to set the vsync callback.
+ *
+ * This function should do whatever is necessary for this integration to ensure that kbase_pm_report_vsync is
+ * called appropriately.
+ *
+ * This function will need porting as part of the integration for a device.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_register_vsync_callback(struct kbase_device *kbdev);
+
+/** Free any resources that kbase_pm_register_vsync_callback allocated.
+ *
+ * This function should perform any cleanup required from the call to kbase_pm_register_vsync_callback.
+ * No call backs should occur after this function has returned.
+ *
+ * This function will need porting as part of the integration for a device.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_unregister_vsync_callback(struct kbase_device *kbdev);
+
+/** Determine whether the DVFS system should change the clock speed of the GPU.
+ *
+ * This function should be called regularly by the DVFS system to check whether the clock speed of the GPU needs
+ * updating. It will return one of three enumerated values of kbase_pm_dvfs_action:
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ * @retval KBASE_PM_DVFS_NOP The clock does not need changing
+ * @retval KBASE_PM_DVFS_CLOCK_UP, The clock frequency should be increased if possible.
+ * @retval KBASE_PM_DVFS_CLOCK_DOWN The clock frequency should be decreased if possible.
+ */
+kbase_pm_dvfs_action kbase_pm_get_dvfs_action(struct kbase_device *kbdev);
+
+/** Mark that the GPU cycle counter is needed, if the caller is the first caller
+ * then the GPU cycle counters will be enabled.
+ *
+ * The GPU must be powered when calling this function (i.e. @ref kbase_pm_context_active must have been called).
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+
+void kbase_pm_request_gpu_cycle_counter(struct kbase_device *kbdev);
+
+/** Mark that the GPU cycle counter is no longer in use, if the caller is the last
+ * caller then the GPU cycle counters will be disabled. A request must have been made
+ * before a call to this.
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+
+void kbase_pm_release_gpu_cycle_counter(struct kbase_device *kbdev);
+
+/** Enables access to the GPU registers before power management has powered up the GPU
+ * with kbase_pm_powerup().
+ *
+ * Access to registers should be done using kbase_os_reg_read/write() at this stage,
+ * not kbase_reg_read/write().
+ *
+ * This results in the power management callbacks provided in the driver configuration
+ * to get called to turn on power and/or clocks to the GPU.
+ * See @ref kbase_pm_callback_conf.
+ *
+ * This should only be used before power management is powered up with kbase_pm_powerup()
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_register_access_enable(struct kbase_device *kbdev);
+
+/** Disables access to the GPU registers enabled earlier by a call to
+ * kbase_pm_register_access_enable().
+ *
+ * This results in the power management callbacks provided in the driver configuration
+ * to get called to turn off power and/or clocks to the GPU.
+ * See @ref kbase_pm_callback_conf
+ *
+ * This should only be used before power management is powered up with kbase_pm_powerup()
+ *
+ * @param kbdev The kbase device structure for the device (must be a valid pointer)
+ */
+void kbase_pm_register_access_disable(struct kbase_device *kbdev);
+
+#endif /* _KBASE_PM_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_pm_always_on.c
+ * "Always on" power management policy
+ */
+
+#include <osk/mali_osk.h>
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_kbase_pm.h>
+
+
+/** Function to handle a GPU state change for the always_on power policy.
+ *
+ * This function is called whenever the GPU has transitioned to another state. It first checks that the transition is
+ * complete and then moves the state machine to the next state.
+ *
+ * @param kbdev The kbase device structure for the device
+ */
+static void always_on_state_changed(kbase_device *kbdev)
+{
+ kbasep_pm_policy_always_on *data = &kbdev->pm.policy_data.always_on;
+
+ switch(data->state)
+ {
+ case KBASEP_PM_ALWAYS_ON_STATE_POWERING_UP:
+ if (kbase_pm_get_pwr_active(kbdev))
+ {
+ /* Cores still transitioning */
+ return;
+ }
+ /* All cores have transitioned, inform the OS */
+ kbase_pm_power_up_done(kbdev);
+ data->state = KBASEP_PM_ALWAYS_ON_STATE_POWERED_UP;
+
+ break;
+ case KBASEP_PM_ALWAYS_ON_STATE_POWERING_DOWN:
+ if (kbase_pm_get_pwr_active(kbdev))
+ {
+ /* Cores still transitioning */
+ return;
+ }
+ /* All cores have transitioned, turn the clock and interrupts off */
+ kbase_pm_disable_interrupts(kbdev);
+ kbase_pm_clock_off(kbdev);
+
+ /* Inform the OS */
+ kbase_pm_power_down_done(kbdev);
+
+ data->state = KBASEP_PM_ALWAYS_ON_STATE_POWERED_DOWN;
+
+ break;
+ case KBASEP_PM_ALWAYS_ON_STATE_CHANGING_POLICY:
+ if (kbase_pm_get_pwr_active(kbdev))
+ {
+ /* Cores still transitioning */
+ return;
+ }
+ /* All cores have transitioned, inform the system we can change policy*/
+ kbase_pm_change_policy(kbdev);
+
+ break;
+ default:
+ break;
+ }
+}
+
+/** Function to handle the @ref KBASE_PM_EVENT_SYSTEM_SUSPEND message for the always_on power policy.
+ *
+ * This function is called when a @ref KBASE_PM_EVENT_SYSTEM_SUSPEND message is received. It instructs the GPU to turn off
+ * all cores.
+ *
+ * @param kbdev The kbase device structure for the device
+ */
+static void always_on_suspend(kbase_device *kbdev)
+{
+ u64 cores;
+
+ /* Inform the system that the transition has started */
+ kbase_pm_power_transitioning(kbdev);
+
+ /* Turn the cores off */
+ cores = kbase_pm_get_present_cores(kbdev, KBASE_PM_CORE_SHADER);
+ kbase_pm_invoke_power_down(kbdev, KBASE_PM_CORE_SHADER, cores);
+
+ cores = kbase_pm_get_present_cores(kbdev, KBASE_PM_CORE_TILER);
+ kbase_pm_invoke_power_down(kbdev, KBASE_PM_CORE_TILER, cores);
+
+ kbase_pm_check_transitions(kbdev);
+
+ kbdev->pm.policy_data.always_on.state = KBASEP_PM_ALWAYS_ON_STATE_POWERING_DOWN;
+
+ /* Ensure that the OS is informed even if we didn't do anything */
+ always_on_state_changed(kbdev);
+}
+
+/** Function to handle the @ref KBASE_PM_EVENT_SYSTEM_RESUME message for the always_on power policy.
+ *
+ * This function is called when a @ref KBASE_PM_EVENT_SYSTEM_RESUME message is received. It instructs the GPU to turn on all
+ * the cores.
+ *
+ * @param kbdev The kbase device structure for the device
+ */
+static void always_on_resume(kbase_device *kbdev)
+{
+ u64 cores;
+
+ /* Inform the system that the transition has started */
+ kbase_pm_power_transitioning(kbdev);
+
+ /* Turn the clock on */
+ kbase_pm_clock_on(kbdev);
+ /* Enable interrupts */
+ kbase_pm_enable_interrupts(kbdev);
+
+ /* Turn the cores on */
+ cores = kbase_pm_get_present_cores(kbdev, KBASE_PM_CORE_SHADER);
+ kbase_pm_invoke_power_up(kbdev, KBASE_PM_CORE_SHADER, cores);
+
+ cores = kbase_pm_get_present_cores(kbdev, KBASE_PM_CORE_TILER);
+ kbase_pm_invoke_power_up(kbdev, KBASE_PM_CORE_TILER, cores);
+
+ kbase_pm_check_transitions(kbdev);
+
+ kbdev->pm.policy_data.always_on.state = KBASEP_PM_ALWAYS_ON_STATE_POWERING_UP;
+
+ /* Ensure that the OS is informed even if we didn't do anything */
+ always_on_state_changed(kbdev);
+}
+
+/** The event callback function for the always_on power policy.
+ *
+ * This function is called to handle the events for the power policy. It calls the relevant handler function depending
+ * on the type of the event.
+ *
+ * @param kbdev The kbase device structure for the device
+ * @param event The event that should be processed
+ */
+static void always_on_event(kbase_device *kbdev, kbase_pm_event event)
+{
+ kbasep_pm_policy_always_on *data = &kbdev->pm.policy_data.always_on;
+
+ switch(event)
+ {
+ case KBASE_PM_EVENT_SYSTEM_SUSPEND:
+ always_on_suspend(kbdev);
+ break;
+ case KBASE_PM_EVENT_POLICY_INIT: /* Init is the same as resume for this policy */
+ case KBASE_PM_EVENT_SYSTEM_RESUME:
+ always_on_resume(kbdev);
+ break;
+ case KBASE_PM_EVENT_GPU_STATE_CHANGED:
+ always_on_state_changed(kbdev);
+ break;
+ case KBASE_PM_EVENT_POLICY_CHANGE:
+ if (data->state == KBASEP_PM_ALWAYS_ON_STATE_POWERED_UP ||
+ data->state == KBASEP_PM_ALWAYS_ON_STATE_POWERED_DOWN)
+ {
+ kbase_pm_change_policy(kbdev);
+ }
+ else
+ {
+ data->state = KBASEP_PM_ALWAYS_ON_STATE_CHANGING_POLICY;
+ }
+ break;
+ case KBASE_PM_EVENT_GPU_ACTIVE:
+ case KBASE_PM_EVENT_GPU_IDLE:
+ case KBASE_PM_EVENT_CHANGE_GPU_STATE:
+ /* Not used - the GPU is always kept on */
+ break;
+ default:
+ /* Unrecognised event - this should never happen */
+ OSK_ASSERT(0);
+ }
+}
+
+/** Initialize the always_on power policy
+ *
+ * This sets up the private @ref kbase_pm_device_data.policy_data field of the device for use with the always_on power
+ * policy.
+ *
+ * @param kbdev The kbase device structure for the device
+ */
+static void always_on_init(kbase_device *kbdev)
+{
+ kbasep_pm_policy_always_on *data = &kbdev->pm.policy_data.always_on;
+
+ data->state = KBASEP_PM_ALWAYS_ON_STATE_POWERING_UP;
+}
+
+/** Terminate the always_on power policy
+ *
+ * This frees the resources that were allocated by @ref always_on_init.
+ *
+ * @param kbdev The kbase device structure for the device
+ */
+static void always_on_term(kbase_device *kbdev)
+{
+ CSTD_UNUSED(kbdev);
+}
+
+/** The @ref kbase_pm_policy structure for the always_on power policy
+ *
+ * This is the extern structure that defines the always_on power policy's callback and name.
+ */
+const kbase_pm_policy kbase_pm_always_on_policy_ops =
+{
+ "always_on", /* name */
+ always_on_init, /* init */
+ always_on_term, /* term */
+ always_on_event, /* event */
+};
+
+KBASE_EXPORT_TEST_API(kbase_pm_always_on_policy_ops)
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_pm_always_on.h
+ * "Always on" power management policy
+ */
+
+#ifndef MALI_KBASE_PM_ALWAYS_ON_H
+#define MALI_KBASE_PM_ALWAYS_ON_H
+
+/** The states that the always_on policy can enter.
+ *
+ * The diagram below should the states that the always_on policy can enter and the transitions that can occur between
+ * the states:
+ *
+ * @dot
+ * digraph always_on_states {
+ * node [fontsize=10];
+ * edge [fontsize=10];
+ *
+ * POWERING_UP [label="STATE_POWERING_UP"
+ * URL="\ref kbasep_pm_always_on_state.KBASEP_PM_ALWAYS_ON_STATE_POWERING_UP"];
+ * POWERING_DOWN [label="STATE_POWERING_DOWN"
+ * URL="\ref kbasep_pm_always_on_state.KBASEP_PM_ALWAYS_ON_STATE_POWERING_DOWN"];
+ * POWERED_UP [label="STATE_POWERED_UP"
+ * URL="\ref kbasep_pm_always_on_state.KBASEP_PM_ALWAYS_ON_STATE_POWERED_UP"];
+ * POWERED_DOWN [label="STATE_POWERED_DOWN"
+ * URL="\ref kbasep_pm_always_on_state.KBASEP_PM_ALWAYS_ON_STATE_POWERED_DOWN"];
+ * CHANGING_POLICY [label="STATE_CHANGING_POLICY"
+ * URL="\ref kbasep_pm_always_on_state.KBASEP_PM_ALWAYS_ON_STATE_CHANGING_POLICY"];
+ *
+ * init [label="init" URL="\ref KBASE_PM_EVENT_INIT"];
+ * change_policy [label="change_policy" URL="\ref kbase_pm_change_policy"];
+ *
+ * init -> POWERING_UP [ label = "Policy init" ];
+ *
+ * POWERING_UP -> POWERED_UP [label = "Power state change" URL="\ref KBASE_PM_EVENT_STATE_CHANGED"];
+ * POWERING_DOWN -> POWERED_DOWN [label = "Power state change" URL="\ref KBASE_PM_EVENT_STATE_CHANGED"];
+ * CHANGING_POLICY -> change_policy [label = "Power state change" URL="\ref KBASE_PM_EVENT_STATE_CHANGED"];
+ *
+ * POWERED_UP -> POWERING_DOWN [label = "Suspend" URL="\ref KBASE_PM_EVENT_SUSPEND"];
+ *
+ * POWERED_DOWN -> POWERING_UP [label = "Resume" URL="\ref KBASE_PM_EVENT_RESUME"];
+ *
+ * POWERING_UP -> CHANGING_POLICY [label = "Change policy" URL="\ref KBASE_PM_EVENT_CHANGE_POLICY"];
+ * POWERING_DOWN -> CHANGING_POLICY [label = "Change policy" URL="\ref KBASE_PM_EVENT_CHANGE_POLICY"];
+ * POWERED_UP -> change_policy [label = "Change policy" URL="\ref KBASE_PM_EVENT_CHANGE_POLICY"];
+ * POWERED_DOWN -> change_policy [label = "Change policy" URL="\ref KBASE_PM_EVENT_CHANGE_POLICY"];
+ * }
+ * @enddot
+ */
+typedef enum kbasep_pm_always_on_state
+{
+ KBASEP_PM_ALWAYS_ON_STATE_POWERING_UP, /**< The GPU is powering up */
+ KBASEP_PM_ALWAYS_ON_STATE_POWERING_DOWN, /**< The GPU is powering down */
+ KBASEP_PM_ALWAYS_ON_STATE_POWERED_UP, /**< The GPU is powered up and jobs can execute */
+ KBASEP_PM_ALWAYS_ON_STATE_POWERED_DOWN, /**< The GPU is powered down and the system can suspend */
+ KBASEP_PM_ALWAYS_ON_STATE_CHANGING_POLICY /**< The power policy is about to change */
+} kbasep_pm_always_on_state;
+
+/** Private structure for policy instance data.
+ *
+ * This contains data that is private to the particular power policy that is active.
+ */
+typedef struct kbasep_pm_policy_always_on
+{
+ kbasep_pm_always_on_state state; /**< The current state of the policy */
+} kbasep_pm_policy_always_on;
+
+#endif
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_pm_demand.c
+ * A simple demand based power management policy
+ */
+
+#include <osk/mali_osk.h>
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_kbase_pm.h>
+
+/* Forward declaration for state change function, as it is required by
+ * the power up and down functions */
+static void demand_state_changed(kbase_device *kbdev);
+
+/** Turns the cores on.
+ *
+ * This function turns all the cores of the GPU on.
+ */
+static void demand_power_up(kbase_device *kbdev)
+{
+ /* Inform the system that the transition has started */
+ kbase_pm_power_transitioning(kbdev);
+
+ /* Turn clocks and interrupts on */
+ kbase_pm_clock_on(kbdev);
+ kbase_pm_enable_interrupts(kbdev);
+
+ kbase_pm_check_transitions(kbdev);
+
+ kbdev->pm.policy_data.demand.state = KBASEP_PM_DEMAND_STATE_POWERING_UP;
+}
+
+/** Turn the cores off.
+ *
+ * This function turns all the cores of the GPU off.
+ */
+static void demand_power_down(kbase_device *kbdev)
+{
+ u64 cores;
+
+ /* Inform the system that the transition has started */
+ kbase_pm_power_transitioning(kbdev);
+
+ /* Turn the cores off */
+ cores = kbase_pm_get_present_cores(kbdev, KBASE_PM_CORE_SHADER);
+ kbase_pm_invoke_power_down(kbdev, KBASE_PM_CORE_SHADER, cores);
+
+ cores = kbase_pm_get_present_cores(kbdev, KBASE_PM_CORE_TILER);
+ kbase_pm_invoke_power_down(kbdev, KBASE_PM_CORE_TILER, cores);
+
+ kbdev->pm.policy_data.demand.state = KBASEP_PM_DEMAND_STATE_POWERING_DOWN;
+
+ kbase_pm_check_transitions(kbdev);
+}
+
+/** Turn some cores on/off.
+ *
+ * This function turns on/off the cores needed by the scheduler.
+ */
+static void demand_change_gpu_state(kbase_device *kbdev)
+{
+ /* Update the bitmap of the cores we need */
+ u64 new_shader_desired = kbdev->shader_needed_bitmap | kbdev->shader_inuse_bitmap;
+ u64 new_tiler_desired = kbdev->tiler_needed_bitmap | kbdev->tiler_inuse_bitmap;
+
+ if ( kbdev->pm.desired_shader_state != new_shader_desired )
+ {
+ KBASE_TRACE_ADD( kbdev, PM_CORES_CHANGE_DESIRED, NULL, NULL, 0u, (u32)new_shader_desired );
+ }
+
+ kbdev->pm.desired_shader_state = new_shader_desired;
+ kbdev->pm.desired_tiler_state = new_tiler_desired;
+
+ kbase_pm_check_transitions(kbdev);
+}
+
+/** Function to handle a GPU state change for the demand power policy
+ *
+ * This function is called whenever the GPU has transitioned to another state. It first checks that the transition is
+ * complete and then moves the state machine to the next state.
+ */
+static void demand_state_changed(kbase_device *kbdev)
+{
+ kbasep_pm_policy_demand *data = &kbdev->pm.policy_data.demand;
+
+ switch(data->state) {
+ case KBASEP_PM_DEMAND_STATE_CHANGING_POLICY:
+ case KBASEP_PM_DEMAND_STATE_POWERING_UP:
+ case KBASEP_PM_DEMAND_STATE_POWERING_DOWN:
+ if (kbase_pm_get_pwr_active(kbdev)) {
+ /* Cores are still transitioning - ignore the event */
+ return;
+ }
+ break;
+ default:
+ /* Must not call kbase_pm_get_pwr_active here as the clock may be turned off */
+ break;
+ }
+
+ switch(data->state)
+ {
+ case KBASEP_PM_DEMAND_STATE_CHANGING_POLICY:
+ /* Signal power events before switching the policy */
+ kbase_pm_power_up_done(kbdev);
+ kbase_pm_power_down_done(kbdev);
+ kbase_pm_change_policy(kbdev);
+ break;
+ case KBASEP_PM_DEMAND_STATE_POWERING_UP:
+ data->state = KBASEP_PM_DEMAND_STATE_POWERED_UP;
+ kbase_pm_power_up_done(kbdev);
+ /* State changed, try to run jobs */
+ KBASE_TRACE_ADD( kbdev, PM_JOB_SUBMIT_AFTER_POWERING_UP, NULL, NULL, 0u, 0 );
+ kbase_js_try_run_jobs(kbdev);
+ break;
+ case KBASEP_PM_DEMAND_STATE_POWERING_DOWN:
+ data->state = KBASEP_PM_DEMAND_STATE_POWERED_DOWN;
+ /* Disable interrupts and turn the clock off */
+ kbase_pm_disable_interrupts(kbdev);
+ kbase_pm_clock_off(kbdev);
+ kbase_pm_power_down_done(kbdev);
+ break;
+ case KBASEP_PM_DEMAND_STATE_POWERED_UP:
+ /* Core states may have been changed, try to run jobs */
+ KBASE_TRACE_ADD( kbdev, PM_JOB_SUBMIT_AFTER_POWERED_UP, NULL, NULL, 0u, 0 );
+ kbase_js_try_run_jobs(kbdev);
+ break;
+ default:
+ break;
+ }
+}
+
+/** The event callback function for the demand power policy.
+ *
+ * This function is called to handle the events for the power policy. It calls the relevant handler function depending
+ * on the type of the event.
+ *
+ * @param kbdev The kbase device structure for the device
+ * @param event The event that should be processed
+ */
+static void demand_event(kbase_device *kbdev, kbase_pm_event event)
+{
+ kbasep_pm_policy_demand *data = &kbdev->pm.policy_data.demand;
+
+ switch(event)
+ {
+ case KBASE_PM_EVENT_POLICY_INIT:
+ demand_power_up(kbdev);
+ break;
+ case KBASE_PM_EVENT_POLICY_CHANGE:
+ if (data->state == KBASEP_PM_DEMAND_STATE_POWERED_UP ||
+ data->state == KBASEP_PM_DEMAND_STATE_POWERED_DOWN)
+ {
+ kbase_pm_change_policy(kbdev);
+ }
+ else
+ {
+ data->state = KBASEP_PM_DEMAND_STATE_CHANGING_POLICY;
+ }
+ break;
+ case KBASE_PM_EVENT_SYSTEM_RESUME:
+ case KBASE_PM_EVENT_GPU_ACTIVE:
+ switch (data->state)
+ {
+ case KBASEP_PM_DEMAND_STATE_POWERING_UP:
+ break;
+ case KBASEP_PM_DEMAND_STATE_POWERED_UP:
+ kbase_pm_power_up_done(kbdev);
+ break;
+ default:
+ demand_power_up(kbdev);
+ }
+ break;
+ case KBASE_PM_EVENT_SYSTEM_SUSPEND:
+ case KBASE_PM_EVENT_GPU_IDLE:
+ switch (data->state)
+ {
+ case KBASEP_PM_DEMAND_STATE_POWERING_DOWN:
+ break;
+ case KBASEP_PM_DEMAND_STATE_POWERED_DOWN:
+ kbase_pm_power_down_done(kbdev);
+ break;
+ default:
+ demand_power_down(kbdev);
+ }
+ break;
+ case KBASE_PM_EVENT_CHANGE_GPU_STATE:
+ if (data->state != KBASEP_PM_DEMAND_STATE_POWERED_DOWN &&
+ data->state != KBASEP_PM_DEMAND_STATE_POWERING_DOWN)
+ {
+ demand_change_gpu_state(kbdev);
+ }
+ break;
+ case KBASE_PM_EVENT_GPU_STATE_CHANGED:
+ demand_state_changed(kbdev);
+ break;
+ default:
+ /* unrecognized event, should never happen */
+ OSK_ASSERT(0);
+ }
+}
+
+/** Initialize the demand power policy.
+ *
+ * This sets up the private @ref kbase_pm_device_data.policy_data field of the device for use with the demand power
+ * policy.
+ *
+ * @param kbdev The kbase device structure for the device
+ */
+static void demand_init(kbase_device *kbdev)
+{
+ kbdev->pm.policy_data.demand.state = KBASEP_PM_DEMAND_STATE_POWERED_UP;
+}
+
+/** Terminate the demand power policy.
+ *
+ * This frees the resources that were allocated by @ref demand_init.
+ *
+ * @param kbdev The kbase device structure for the device
+ */
+static void demand_term(kbase_device *kbdev)
+{
+ CSTD_UNUSED(kbdev);
+}
+
+/** The @ref kbase_pm_policy structure for the demand power policy.
+ *
+ * This is the static structure that defines the demand power policy's callback and name.
+ */
+const kbase_pm_policy kbase_pm_demand_policy_ops =
+{
+ "demand", /* name */
+ demand_init, /* init */
+ demand_term, /* term */
+ demand_event, /* event */
+};
+
+
+KBASE_EXPORT_TEST_API(kbase_pm_demand_policy_ops)
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_pm_demand.h
+ * A simple demand based power management policy
+ */
+
+#ifndef MALI_KBASE_PM_DEMAND_H
+#define MALI_KBASE_PM_DEMAND_H
+
+/** The states that the demand policy can enter.
+ *
+ * The diagram below should the states that the demand policy can enter and the transitions that can occur between the
+ * states:
+ *
+ * @dot
+ * digraph demand_states {
+ * node [fontsize=10];
+ * edge [fontsize=10];
+ *
+ * POWERING_UP [label="STATE_POWERING_UP"
+ * URL="\ref kbasep_pm_demand_state.KBASEP_PM_DEMAND_STATE_POWERING_UP"];
+ * POWERING_DOWN [label="STATE_POWERING_DOWN"
+ * URL="\ref kbasep_pm_demand_state.KBASEP_PM_DEMAND_STATE_POWERING_DOWN"];
+ * POWERED_UP [label="STATE_POWERED_UP"
+ * URL="\ref kbasep_pm_demand_state.KBASEP_PM_DEMAND_STATE_POWERED_UP"];
+ * POWERED_DOWN [label="STATE_POWERED_DOWN"
+ * URL="\ref kbasep_pm_demand_state.KBASEP_PM_DEMAND_STATE_POWERED_DOWN"];
+ * CHANGING_POLICY [label="STATE_CHANGING_POLICY"
+ * URL="\ref kbasep_pm_demand_state.KBASEP_PM_DEMAND_STATE_CHANGING_POLICY"];
+ *
+ * init [label="init" URL="\ref KBASE_PM_EVENT_INIT"];
+ * change_policy [label="change_policy" URL="\ref kbase_pm_change_policy"];
+ *
+ * init -> POWERING_UP [ label = "Policy init" ];
+ *
+ * POWERING_UP -> POWERED_UP [label = "Power state change" URL="\ref KBASE_PM_EVENT_STATE_CHANGED"];
+ * POWERING_DOWN -> POWERED_DOWN [label = "Power state change" URL="\ref KBASE_PM_EVENT_STATE_CHANGED"];
+ * CHANGING_POLICY -> change_policy [label = "Power state change" URL="\ref KBASE_PM_EVENT_STATE_CHANGED"];
+ *
+ * POWERED_UP -> POWERING_DOWN [label = "GPU Idle" URL="\ref KBASE_PM_EVENT_GPU_IDLE"];
+ * POWERING_UP -> POWERING_DOWN [label = "GPU Idle" URL="\ref KBASE_PM_EVENT_GPU_IDLE"];
+ *
+ * POWERED_DOWN -> POWERING_UP [label = "GPU Active" URL="\ref KBASE_PM_EVENT_GPU_ACTIVE"];
+ * POWERING_DOWN -> POWERING_UP [label = "GPU Active" URL="\ref KBASE_PM_EVENT_GPU_ACTIVE"];
+
+ * POWERING_UP -> CHANGING_POLICY [label = "Change policy" URL="\ref KBASE_PM_EVENT_CHANGE_POLICY"];
+ * POWERING_DOWN -> CHANGING_POLICY [label = "Change policy" URL="\ref KBASE_PM_EVENT_CHANGE_POLICY"];
+ * POWERED_UP -> change_policy [label = "Change policy" URL="\ref KBASE_PM_EVENT_CHANGE_POLICY"];
+ * POWERED_DOWN -> change_policy [label = "Change policy" URL="\ref KBASE_PM_EVENT_CHANGE_POLICY"];
+ * }
+ * @enddot
+ */
+typedef enum kbasep_pm_demand_state
+{
+ KBASEP_PM_DEMAND_STATE_POWERING_UP, /**< The GPU is powering up */
+ KBASEP_PM_DEMAND_STATE_POWERED_UP, /**< The GPU is powered up and jobs can execute */
+ KBASEP_PM_DEMAND_STATE_POWERING_DOWN, /**< The GPU is powering down */
+ KBASEP_PM_DEMAND_STATE_POWERED_DOWN, /**< The GPU is powered down */
+ KBASEP_PM_DEMAND_STATE_CHANGING_POLICY /**< The power policy is about to change */
+} kbasep_pm_demand_state;
+
+/** Private structure for policy instance data.
+ *
+ * This contains data that is private to the particular power policy that is active.
+ */
+typedef struct kbasep_pm_policy_demand
+{
+ kbasep_pm_demand_state state; /**< The current state of the policy */
+} kbasep_pm_policy_demand;
+
+#endif
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_pm_driver.c
+ * Base kernel Power Management hardware control
+ */
+
+#include <osk/mali_osk.h>
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_midg_regmap.h>
+#include <kbase/src/common/mali_kbase_gator.h>
+#include <kbase/src/common/mali_kbase_pm.h>
+
+#if MALI_MOCK_TEST
+#define MOCKABLE(function) function##_original
+#else
+#define MOCKABLE(function) function
+#endif /* MALI_MOCK_TEST */
+
+/** Number of milliseconds before we time out on a reset */
+#define RESET_TIMEOUT 500
+
+/** Actions that can be performed on a core.
+ *
+ * This enumeration is private to the file. Its values are set to allow @ref core_type_to_reg function,
+ * which decodes this enumeration, to be simpler and more efficient.
+ */
+typedef enum kbasep_pm_action
+{
+ ACTION_PRESENT = 0,
+ ACTION_READY = (SHADER_READY_LO - SHADER_PRESENT_LO),
+ ACTION_PWRON = (SHADER_PWRON_LO - SHADER_PRESENT_LO),
+ ACTION_PWROFF = (SHADER_PWROFF_LO - SHADER_PRESENT_LO),
+ ACTION_PWRTRANS = (SHADER_PWRTRANS_LO - SHADER_PRESENT_LO),
+ ACTION_PWRACTIVE = (SHADER_PWRACTIVE_LO - SHADER_PRESENT_LO)
+} kbasep_pm_action;
+
+/** Decode a core type and action to a register.
+ *
+ * Given a core type (defined by @ref kbase_pm_core_type) and an action (defined by @ref kbasep_pm_action) this
+ * function will return the register offset that will perform the action on the core type. The register returned is
+ * the \c _LO register and an offset must be applied to use the \c _HI register.
+ *
+ * @param core_type The type of core
+ * @param action The type of action
+ *
+ * @return The register offset of the \c _LO register that performs an action of type \c action on a core of type \c
+ * core_type.
+ */
+static u32 core_type_to_reg(kbase_pm_core_type core_type, kbasep_pm_action action)
+{
+ return core_type + action;
+}
+
+/** Invokes an action on a core set
+ *
+ * This function performs the action given by \c action on a set of cores of a type given by \c core_type. It is a
+ * static function used by @ref kbase_pm_invoke_power_up and @ref kbase_pm_invoke_power_down.
+ *
+ * @param kbdev The kbase device structure of the device
+ * @param core_type The type of core that the action should be performed on
+ * @param cores A bit mask of cores to perform the action on (low 32 bits)
+ * @param action The action to perform on the cores
+ */
+STATIC void kbase_pm_invoke(kbase_device *kbdev, kbase_pm_core_type core_type, u64 cores, kbasep_pm_action action)
+{
+ u32 reg;
+ u32 lo = cores & 0xFFFFFFFF;
+ u32 hi = (cores >> 32) & 0xFFFFFFFF;
+
+ reg = core_type_to_reg(core_type, action);
+
+ OSK_ASSERT(reg);
+#if MALI_GATOR_SUPPORT
+ if (cores)
+ {
+ if (action == ACTION_PWRON )
+ {
+ kbase_trace_mali_pm_power_on(core_type, cores);
+ }
+ else if ( action == ACTION_PWROFF )
+ {
+ kbase_trace_mali_pm_power_off(core_type, cores);
+ }
+ }
+#endif
+ /* Tracing */
+ if ( cores != 0 && core_type == KBASE_PM_CORE_SHADER )
+ {
+ if (action == ACTION_PWRON )
+ {
+ KBASE_TRACE_ADD( kbdev, PM_PWRON, NULL, NULL, 0u, lo );
+ }
+ else if ( action == ACTION_PWROFF )
+ {
+ KBASE_TRACE_ADD( kbdev, PM_PWROFF, NULL, NULL, 0u, lo );
+ }
+ }
+
+ if (lo != 0)
+ {
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(reg), lo, NULL);
+ }
+ if (hi != 0)
+ {
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(reg+4), hi, NULL);
+ }
+}
+
+void kbase_pm_invoke_power_up(kbase_device *kbdev, kbase_pm_core_type type, u64 cores)
+{
+ OSK_ASSERT( kbdev != NULL );
+
+ switch(type)
+ {
+ case KBASE_PM_CORE_SHADER:
+ {
+ u64 prev_desired_shader = kbdev->pm.desired_shader_state;
+ kbdev->pm.desired_shader_state |= cores;
+ if ( prev_desired_shader != kbdev->pm.desired_shader_state )
+ {
+ KBASE_TRACE_ADD( kbdev, PM_CORES_CHANGE_DESIRED_ON_POWERUP, NULL, NULL, 0u, (u32)kbdev->pm.desired_shader_state );
+ }
+ }
+ break;
+ case KBASE_PM_CORE_TILER:
+ kbdev->pm.desired_tiler_state |= cores;
+ break;
+ default:
+ OSK_ASSERT(0);
+ }
+}
+KBASE_EXPORT_TEST_API(kbase_pm_invoke_power_up)
+
+void kbase_pm_invoke_power_down(kbase_device *kbdev, kbase_pm_core_type type, u64 cores)
+{
+ OSK_ASSERT( kbdev != NULL );
+
+ switch(type)
+ {
+ case KBASE_PM_CORE_SHADER:
+ {
+ u64 prev_desired_shader = kbdev->pm.desired_shader_state;
+ kbdev->pm.desired_shader_state &= ~cores;
+ if ( prev_desired_shader != kbdev->pm.desired_shader_state )
+ {
+ KBASE_TRACE_ADD( kbdev, PM_CORES_CHANGE_DESIRED_ON_POWERDOWN, NULL, NULL, 0u, (u32)kbdev->pm.desired_shader_state );
+ }
+ }
+ break;
+ case KBASE_PM_CORE_TILER:
+ kbdev->pm.desired_tiler_state &= ~cores;
+ break;
+ default:
+ OSK_ASSERT(0);
+ }
+}
+KBASE_EXPORT_TEST_API(kbase_pm_invoke_power_down)
+/** Get information about a core set
+ *
+ * This function gets information (chosen by \c action) about a set of cores of a type given by \c core_type. It is a
+ * static function used by @ref kbase_pm_get_present_cores, @ref kbase_pm_get_active_cores, @ref
+ * kbase_pm_get_trans_cores and @ref kbase_pm_get_ready_cores.
+ *
+ * @param kbdev The kbase device structure of the device
+ * @param core_type The type of core that the should be queried
+ * @param action The property of the cores to query
+ *
+ * @return A bit mask specifying the state of the cores
+ */
+static u64 kbase_pm_get_state(kbase_device *kbdev, kbase_pm_core_type core_type, kbasep_pm_action action)
+{
+ u32 reg;
+ u32 lo, hi;
+
+ reg = core_type_to_reg(core_type, action);
+
+ OSK_ASSERT(reg);
+
+ lo = kbase_reg_read(kbdev, GPU_CONTROL_REG(reg), NULL);
+ hi = kbase_reg_read(kbdev, GPU_CONTROL_REG(reg+4), NULL);
+
+ return (((u64)hi) << 32) | ((u64)lo);
+}
+
+void kbasep_pm_read_present_cores(kbase_device *kbdev)
+{
+ kbdev->shader_present_bitmap = kbase_pm_get_state(kbdev, KBASE_PM_CORE_SHADER, ACTION_PRESENT);
+ kbdev->tiler_present_bitmap = kbase_pm_get_state(kbdev, KBASE_PM_CORE_TILER, ACTION_PRESENT);
+ kbdev->l2_present_bitmap = kbase_pm_get_state(kbdev, KBASE_PM_CORE_L2, ACTION_PRESENT);
+ kbdev->l3_present_bitmap = kbase_pm_get_state(kbdev, KBASE_PM_CORE_L3, ACTION_PRESENT);
+
+ kbdev->shader_inuse_bitmap = 0;
+ kbdev->tiler_inuse_bitmap = 0;
+ kbdev->shader_needed_bitmap = 0;
+ kbdev->tiler_needed_bitmap = 0;
+ kbdev->shader_available_bitmap = 0;
+ kbdev->tiler_available_bitmap = 0;
+
+ OSK_MEMSET(kbdev->shader_needed_cnt, 0, sizeof(kbdev->shader_needed_cnt));
+ OSK_MEMSET(kbdev->shader_needed_cnt, 0, sizeof(kbdev->tiler_needed_cnt));
+}
+KBASE_EXPORT_TEST_API(kbasep_pm_read_present_cores)
+
+/** Get the cores that are present
+ */
+u64 kbase_pm_get_present_cores(kbase_device *kbdev, kbase_pm_core_type type)
+{
+ OSK_ASSERT( kbdev != NULL );
+
+ switch(type) {
+ case KBASE_PM_CORE_L3:
+ return kbdev->l3_present_bitmap;
+ break;
+ case KBASE_PM_CORE_L2:
+ return kbdev->l2_present_bitmap;
+ break;
+ case KBASE_PM_CORE_SHADER:
+ return kbdev->shader_present_bitmap;
+ break;
+ case KBASE_PM_CORE_TILER:
+ return kbdev->tiler_present_bitmap;
+ break;
+ }
+ OSK_ASSERT(0);
+ return 0;
+}
+KBASE_EXPORT_TEST_API(kbase_pm_get_present_cores)
+
+/** Get the cores that are "active" (busy processing work)
+ */
+u64 kbase_pm_get_active_cores(kbase_device *kbdev, kbase_pm_core_type type)
+{
+ return kbase_pm_get_state(kbdev, type, ACTION_PWRACTIVE);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_get_active_cores)
+
+/** Get the cores that are transitioning between power states
+ */
+u64 MOCKABLE(kbase_pm_get_trans_cores)(kbase_device *kbdev, kbase_pm_core_type type)
+{
+ return kbase_pm_get_state(kbdev, type, ACTION_PWRTRANS);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_get_trans_cores)
+/** Get the cores that are powered on
+ */
+u64 kbase_pm_get_ready_cores(kbase_device *kbdev, kbase_pm_core_type type)
+{
+ u64 result;
+ result = kbase_pm_get_state(kbdev, type, ACTION_READY);
+
+ if ( type == KBASE_PM_CORE_SHADER )
+ {
+ KBASE_TRACE_ADD( kbdev, PM_CORES_POWERED, NULL, NULL, 0u, (u32)result);
+ }
+ return result;
+}
+KBASE_EXPORT_TEST_API(kbase_pm_get_ready_cores)
+
+/** Is there an active power transition?
+ *
+ * Returns true if there is a power transition in progress, otherwise false.
+ */
+mali_bool MOCKABLE(kbase_pm_get_pwr_active)(kbase_device *kbdev)
+{
+ return ((kbase_reg_read(kbdev, GPU_CONTROL_REG(GPU_STATUS), NULL) & (1<<1)) != 0);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_get_pwr_active)
+
+/** Perform power transitions for a particular core type.
+ *
+ * This function will perform any available power transitions to make the actual hardware state closer to the desired
+ * state. If a core is currently transitioning then changes to the power state of that call cannot be made until the
+ * transition has finished. Cores which are not present in the hardware are ignored if they are specified in the
+ * desired_state bitmask, however the return value will always be 0 in this case.
+ *
+ * @param kbdev The kbase device
+ * @param type The core type to perform transitions for
+ * @param desired_state A bit mask of the desired state of the cores
+ * @param in_use A bit mask of the cores that are currently running jobs.
+ * These cores have to be kept powered up because there are jobs
+ * running (or about to run) on them.
+ * @param[out] available Receives a bit mask of the cores that the job scheduler can use to submit jobs to.
+ * May be NULL if this is not needed.
+ *
+ * @return MALI_TRUE if the desired state has been reached, MALI_FALSE otherwise
+ */
+
+STATIC mali_bool kbase_pm_transition_core_type(kbase_device *kbdev, kbase_pm_core_type type, u64 desired_state,
+ u64 in_use, u64 *available)
+{
+ u64 present;
+ u64 ready;
+ u64 trans;
+ u64 powerup;
+ u64 powerdown;
+
+ /* Get current state */
+ present = kbase_pm_get_present_cores(kbdev, type);
+ trans = kbase_pm_get_trans_cores(kbdev, type);
+ ready = kbase_pm_get_ready_cores(kbdev, type);
+
+ if (available != NULL)
+ {
+ *available = ready & desired_state;
+ }
+
+ /* Update desired state to include the in-use cores. These have to be kept powered up because there are jobs
+ * running or about to run on these cores
+ */
+ desired_state |= in_use;
+
+ /* Workaround for MIDBASE-1258 (L2 usage should be refcounted).
+ * Keep the L2 from being turned off.
+ */
+ if (type == KBASE_PM_CORE_L2)
+ {
+ desired_state = present;
+ }
+
+ if (desired_state == ready && trans == 0)
+ {
+ return MALI_TRUE;
+ }
+
+ /* Restrict the cores to those that are actually present */
+ powerup = desired_state & present;
+ powerdown = (~desired_state) & present;
+
+ /* Restrict to cores that are not already in the desired state */
+ powerup &= ~ready;
+ powerdown &= ready;
+
+ /* Don't transition any cores that are already transitioning */
+ powerup &= ~trans;
+ powerdown &= ~trans;
+
+ /* Perform transitions if any */
+ kbase_pm_invoke(kbdev, type, powerup, ACTION_PWRON);
+ kbase_pm_invoke(kbdev, type, powerdown, ACTION_PWROFF);
+
+ return MALI_FALSE;
+}
+KBASE_EXPORT_TEST_API(kbase_pm_transition_core_type)
+
+/** Determine which caches should be on for a particular core state.
+ *
+ * This function takes a bit mask of the present caches and the cores (or caches) that are attached to the caches that
+ * will be powered. It then computes which caches should be turned on to allow the cores requested to be powered up.
+ *
+ * @param present The bit mask of present caches
+ * @param cores_powered A bit mask of cores (or L2 caches) that are desired to be powered
+ *
+ * @return A bit mask of the caches that should be turned on
+ */
+STATIC u64 get_desired_cache_status(u64 present, u64 cores_powered)
+{
+ u64 desired = 0;
+
+ while (present)
+ {
+ /* Find out which is the highest set bit */
+ u64 bit = 63-osk_clz_64(present);
+ u64 bit_mask = 1ull << bit;
+ /* Create a mask which has all bits from 'bit' upwards set */
+
+ u64 mask = ~(bit_mask-1);
+
+ /* If there are any cores powered at this bit or above (that haven't previously been processed) then we need
+ * this core on */
+ if (cores_powered & mask)
+ {
+ desired |= bit_mask;
+ }
+
+ /* Remove bits from cores_powered and present */
+ cores_powered &= ~mask;
+ present &= ~bit_mask;
+ }
+
+ return desired;
+}
+KBASE_EXPORT_TEST_API(get_desired_cache_status)
+static mali_bool kbasep_pm_unrequest_cores_nolock(kbase_device *kbdev, u64 shader_cores, u64 tiler_cores)
+{
+ mali_bool change_gpu_state = MALI_FALSE;
+
+
+ OSK_ASSERT( kbdev != NULL );
+
+ while (shader_cores)
+ {
+ int bitnum = 63 - osk_clz_64(shader_cores);
+ u64 bit = 1ULL << bitnum;
+ int cnt;
+
+ OSK_ASSERT(kbdev->shader_needed_cnt[bitnum] > 0);
+
+ cnt = --kbdev->shader_needed_cnt[bitnum];
+
+ if (0 == cnt)
+ {
+ kbdev->shader_needed_bitmap &= ~bit;
+ change_gpu_state = MALI_TRUE;
+ }
+
+ shader_cores &= ~bit;
+ }
+
+ if ( change_gpu_state )
+ {
+ KBASE_TRACE_ADD( kbdev, PM_UNREQUEST_CHANGE_SHADER_NEEDED, NULL, NULL, 0u, (u32)kbdev->shader_needed_bitmap );
+ }
+
+ while (tiler_cores)
+ {
+ int bitnum = 63 - osk_clz_64(tiler_cores);
+ u64 bit = 1ULL << bitnum;
+ int cnt;
+
+ OSK_ASSERT(kbdev->tiler_needed_cnt[bitnum] > 0);
+
+ cnt = --kbdev->tiler_needed_cnt[bitnum];
+
+ if (0 == cnt)
+ {
+ kbdev->tiler_needed_bitmap &= ~bit;
+ change_gpu_state = MALI_TRUE;
+ }
+
+ tiler_cores &= ~bit;
+ }
+
+ return change_gpu_state;
+}
+
+mali_error kbase_pm_request_cores(kbase_device *kbdev, u64 shader_cores, u64 tiler_cores)
+{
+ u64 cores;
+
+ mali_bool change_gpu_state = MALI_FALSE;
+
+ OSK_ASSERT( kbdev != NULL );
+
+ osk_spinlock_irq_lock(&kbdev->pm.power_change_lock);
+
+ cores = shader_cores;
+ while (cores)
+ {
+ int bitnum = 63 - osk_clz_64(cores);
+ u64 bit = 1ULL << bitnum;
+
+ int cnt = ++kbdev->shader_needed_cnt[bitnum];
+
+ if (0 == cnt)
+ {
+ /* Wrapped, undo everything we've done so far */
+
+ kbdev->shader_needed_cnt[bitnum]--;
+ kbasep_pm_unrequest_cores_nolock(kbdev, cores ^ shader_cores, 0);
+
+ osk_spinlock_irq_unlock(&kbdev->pm.power_change_lock);
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ if (1 == cnt)
+ {
+ kbdev->shader_needed_bitmap |= bit;
+ change_gpu_state = MALI_TRUE;
+ }
+
+ cores &= ~bit;
+ }
+
+
+ if ( change_gpu_state )
+ {
+ KBASE_TRACE_ADD( kbdev, PM_REQUEST_CHANGE_SHADER_NEEDED, NULL, NULL, 0u, (u32)kbdev->shader_needed_bitmap );
+ }
+
+ cores = tiler_cores;
+ while (cores)
+ {
+ int bitnum = 63 - osk_clz_64(cores);
+ u64 bit = 1ULL << bitnum;
+
+ int cnt = ++kbdev->tiler_needed_cnt[bitnum];
+
+ if (0 == cnt)
+ {
+ /* Wrapped, undo everything we've done so far */
+
+ kbdev->tiler_needed_cnt[bitnum]--;
+ kbasep_pm_unrequest_cores_nolock(kbdev, shader_cores, cores ^ tiler_cores);
+
+ osk_spinlock_irq_unlock(&kbdev->pm.power_change_lock);
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ if (1 == cnt)
+ {
+ kbdev->tiler_needed_bitmap |= bit;
+ change_gpu_state = MALI_TRUE;
+ }
+
+ cores &= ~bit;
+ }
+
+ if (change_gpu_state)
+ {
+ kbase_pm_send_event(kbdev, KBASE_PM_EVENT_CHANGE_GPU_STATE);
+ }
+
+ osk_spinlock_irq_unlock(&kbdev->pm.power_change_lock);
+
+ return MALI_ERROR_NONE;
+}
+KBASE_EXPORT_TEST_API(kbase_pm_request_cores)
+
+void kbase_pm_unrequest_cores(kbase_device *kbdev, u64 shader_cores, u64 tiler_cores)
+{
+ mali_bool change_gpu_state = MALI_FALSE;
+
+
+ OSK_ASSERT( kbdev != NULL );
+
+ osk_spinlock_irq_lock(&kbdev->pm.power_change_lock);
+
+ change_gpu_state = kbasep_pm_unrequest_cores_nolock(kbdev, shader_cores, tiler_cores);
+
+ if (change_gpu_state)
+ {
+ kbase_pm_send_event(kbdev, KBASE_PM_EVENT_CHANGE_GPU_STATE);
+ }
+
+ osk_spinlock_irq_unlock(&kbdev->pm.power_change_lock);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_unrequest_cores)
+
+mali_bool kbase_pm_register_inuse_cores(kbase_device *kbdev, u64 shader_cores, u64 tiler_cores)
+{
+ u64 prev_shader_needed; /* Just for tracing */
+ u64 prev_shader_inuse; /* Just for tracing */
+
+ osk_spinlock_irq_lock(&kbdev->pm.power_change_lock);
+
+ prev_shader_needed = kbdev->shader_needed_bitmap;
+ prev_shader_inuse = kbdev->shader_inuse_bitmap;
+
+ if ((kbdev->shader_available_bitmap & shader_cores) != shader_cores ||
+ (kbdev->tiler_available_bitmap & tiler_cores) != tiler_cores)
+ {
+ osk_spinlock_irq_unlock(&kbdev->pm.power_change_lock);
+ return MALI_FALSE;
+ }
+
+ while (shader_cores)
+ {
+ int bitnum = 63 - osk_clz_64(shader_cores);
+ u64 bit = 1ULL << bitnum;
+ int cnt;
+
+ OSK_ASSERT(kbdev->shader_needed_cnt[bitnum] > 0);
+
+ cnt = --kbdev->shader_needed_cnt[bitnum];
+
+ if (0 == cnt)
+ {
+ kbdev->shader_needed_bitmap &= ~bit;
+ }
+
+ /* shader_inuse_cnt should not overflow because there can only be a
+ * very limited number of jobs on the h/w at one time */
+
+ kbdev->shader_inuse_cnt[bitnum]++;
+ kbdev->shader_inuse_bitmap |= bit;
+
+ shader_cores &= ~bit;
+ }
+
+ if ( prev_shader_needed != kbdev->shader_needed_bitmap )
+ {
+ KBASE_TRACE_ADD( kbdev, PM_REGISTER_CHANGE_SHADER_NEEDED, NULL, NULL, 0u, (u32)kbdev->shader_needed_bitmap );
+ }
+ if ( prev_shader_inuse != kbdev->shader_inuse_bitmap )
+ {
+ KBASE_TRACE_ADD( kbdev, PM_REGISTER_CHANGE_SHADER_INUSE, NULL, NULL, 0u, (u32)kbdev->shader_inuse_bitmap );
+ }
+
+ while (tiler_cores)
+ {
+ int bitnum = 63 - osk_clz_64(tiler_cores);
+ u64 bit = 1ULL << bitnum;
+ int cnt;
+
+ OSK_ASSERT(kbdev->tiler_needed_cnt[bitnum] > 0);
+
+ cnt = --kbdev->tiler_needed_cnt[bitnum];
+
+ if (0 == cnt)
+ {
+ kbdev->tiler_needed_bitmap &= ~bit;
+ }
+
+ /* tiler_inuse_cnt should not overflow because there can only be a
+ * very limited number of jobs on the h/w at one time */
+
+ kbdev->tiler_inuse_cnt[bitnum]++;
+ kbdev->tiler_inuse_bitmap |= bit;
+
+ tiler_cores &= ~bit;
+ }
+
+ osk_spinlock_irq_unlock(&kbdev->pm.power_change_lock);
+
+ return MALI_TRUE;
+}
+KBASE_EXPORT_TEST_API(kbase_pm_register_inuse_cores)
+
+void kbase_pm_release_cores(kbase_device *kbdev, u64 shader_cores, u64 tiler_cores)
+{
+ mali_bool change_gpu_state = MALI_FALSE;
+
+ OSK_ASSERT( kbdev != NULL );
+
+ osk_spinlock_irq_lock(&kbdev->pm.power_change_lock);
+
+ while (shader_cores)
+ {
+ int bitnum = 63 - osk_clz_64(shader_cores);
+ u64 bit = 1ULL << bitnum;
+ int cnt;
+
+ OSK_ASSERT(kbdev->shader_inuse_cnt[bitnum] > 0);
+
+ cnt = --kbdev->shader_inuse_cnt[bitnum];
+
+ if (0 == cnt)
+ {
+ kbdev->shader_inuse_bitmap &= ~bit;
+ change_gpu_state = MALI_TRUE;
+ }
+
+ shader_cores &= ~bit;
+ }
+
+ if ( change_gpu_state )
+ {
+ KBASE_TRACE_ADD( kbdev, PM_RELEASE_CHANGE_SHADER_INUSE, NULL, NULL, 0u, (u32)kbdev->shader_inuse_bitmap );
+ }
+
+ while (tiler_cores)
+ {
+ int bitnum = 63 - osk_clz_64(tiler_cores);
+ u64 bit = 1ULL << bitnum;
+ int cnt;
+
+ OSK_ASSERT(kbdev->tiler_inuse_cnt[bitnum] > 0);
+
+ cnt = --kbdev->tiler_inuse_cnt[bitnum];
+
+ if (0 == cnt)
+ {
+ kbdev->tiler_inuse_bitmap &= ~bit;
+ change_gpu_state = MALI_TRUE;
+ }
+
+ tiler_cores &= ~bit;
+ }
+
+ if (change_gpu_state)
+ {
+ kbase_pm_send_event(kbdev, KBASE_PM_EVENT_CHANGE_GPU_STATE);
+ }
+
+ osk_spinlock_irq_unlock(&kbdev->pm.power_change_lock);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_release_cores)
+
+void MOCKABLE(kbase_pm_check_transitions)(kbase_device *kbdev)
+{
+ mali_bool in_desired_state = MALI_TRUE;
+ u64 desired_l2_state;
+ u64 desired_l3_state;
+ u64 cores_powered;
+ u64 tiler_available_bitmap;
+ u64 shader_available_bitmap;
+
+ OSK_ASSERT( NULL != kbdev );
+
+ osk_spinlock_irq_lock(&kbdev->pm.power_change_lock);
+
+ cores_powered = (kbdev->pm.desired_shader_state | kbdev->pm.desired_tiler_state);
+
+ /* We need to keep the inuse cores powered */
+ cores_powered |= kbdev->shader_inuse_bitmap | kbdev->tiler_inuse_bitmap;
+
+ desired_l2_state = get_desired_cache_status(kbdev->l2_present_bitmap, cores_powered);
+ desired_l3_state = get_desired_cache_status(kbdev->l3_present_bitmap, desired_l2_state);
+
+ in_desired_state &= kbase_pm_transition_core_type(kbdev, KBASE_PM_CORE_L3, desired_l3_state, 0, NULL);
+ in_desired_state &= kbase_pm_transition_core_type(kbdev, KBASE_PM_CORE_L2, desired_l2_state, 0, NULL);
+
+ if (in_desired_state)
+ {
+ in_desired_state &= kbase_pm_transition_core_type(kbdev, KBASE_PM_CORE_TILER,
+ kbdev->pm.desired_tiler_state, kbdev->tiler_inuse_bitmap,
+ &tiler_available_bitmap);
+ in_desired_state &= kbase_pm_transition_core_type(kbdev, KBASE_PM_CORE_SHADER,
+ kbdev->pm.desired_shader_state, kbdev->shader_inuse_bitmap,
+ &shader_available_bitmap);
+
+ /* If we reached the desired state, or we powered off a core, then update the available
+ * This is because:
+ * - powering down happens immediately, so we must make the cores unavailable immediately
+ * - powering up may not bring all cores up together at once, so we must wait until we
+ * reach the desired state before making the cores available */
+ if ( in_desired_state )
+ {
+ if ( kbdev->shader_available_bitmap != shader_available_bitmap )
+ {
+ KBASE_TRACE_ADD( kbdev, PM_CORES_CHANGE_AVAILABLE, NULL, NULL, 0u, (u32)shader_available_bitmap );
+ }
+ kbdev->shader_available_bitmap = shader_available_bitmap;
+ kbdev->tiler_available_bitmap = tiler_available_bitmap;
+ }
+ else
+ {
+ /* Calculate the cores that were previously available and are still available now (i.e.
+ * take account of cores that powered down, but ignore those that powered up) */
+ u64 remaining_shader_available = kbdev->shader_available_bitmap & shader_available_bitmap;
+ u64 remaining_tiler_available = kbdev->tiler_available_bitmap & tiler_available_bitmap;
+ if ( kbdev->shader_available_bitmap != remaining_shader_available )
+ {
+ KBASE_TRACE_ADD( kbdev, PM_CORES_CHANGE_AVAILABLE, NULL, NULL, 0u, (u32)remaining_shader_available );
+ }
+ kbdev->shader_available_bitmap = remaining_shader_available;
+ kbdev->tiler_available_bitmap = remaining_tiler_available;
+ }
+ }
+
+ if (in_desired_state)
+ {
+#if MALI_GATOR_SUPPORT
+ kbase_trace_mali_pm_status(KBASE_PM_CORE_L3, kbase_pm_get_ready_cores(kbdev, KBASE_PM_CORE_L3));
+ kbase_trace_mali_pm_status(KBASE_PM_CORE_L2, kbase_pm_get_ready_cores(kbdev, KBASE_PM_CORE_L2));
+ kbase_trace_mali_pm_status(KBASE_PM_CORE_SHADER, kbase_pm_get_ready_cores(kbdev, KBASE_PM_CORE_SHADER));
+ kbase_trace_mali_pm_status(KBASE_PM_CORE_TILER, kbase_pm_get_ready_cores(kbdev, KBASE_PM_CORE_TILER));
+#endif
+ kbase_pm_send_event(kbdev, KBASE_PM_EVENT_GPU_STATE_CHANGED);
+ }
+
+ osk_spinlock_irq_unlock(&kbdev->pm.power_change_lock);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_check_transitions)
+
+void MOCKABLE(kbase_pm_enable_interrupts)(kbase_device *kbdev)
+{
+
+ OSK_ASSERT( NULL != kbdev );
+ /*
+ * Clear all interrupts,
+ * and unmask them all.
+ */
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_IRQ_CLEAR), GPU_IRQ_REG_ALL, NULL);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_IRQ_MASK), GPU_IRQ_REG_ALL, NULL);
+
+ kbase_reg_write(kbdev, JOB_CONTROL_REG(JOB_IRQ_CLEAR), 0xFFFFFFFF, NULL);
+ kbase_reg_write(kbdev, JOB_CONTROL_REG(JOB_IRQ_MASK), 0xFFFFFFFF, NULL);
+
+ kbase_reg_write(kbdev, MMU_REG(MMU_IRQ_CLEAR), 0xFFFFFFFF, NULL);
+ kbase_reg_write(kbdev, MMU_REG(MMU_IRQ_MASK), 0xFFFFFFFF, NULL);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_enable_interrupts)
+
+void MOCKABLE(kbase_pm_disable_interrupts)(kbase_device *kbdev)
+{
+
+ OSK_ASSERT( NULL != kbdev );
+ /*
+ * Mask all interrupts,
+ * and clear them all.
+ */
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_IRQ_MASK), 0, NULL);
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_IRQ_CLEAR), GPU_IRQ_REG_ALL, NULL);
+
+ kbase_reg_write(kbdev, JOB_CONTROL_REG(JOB_IRQ_MASK), 0, NULL);
+ kbase_reg_write(kbdev, JOB_CONTROL_REG(JOB_IRQ_CLEAR), 0xFFFFFFFF, NULL);
+
+ kbase_reg_write(kbdev, MMU_REG(MMU_IRQ_MASK), 0, NULL);
+ kbase_reg_write(kbdev, MMU_REG(MMU_IRQ_CLEAR), 0xFFFFFFFF, NULL);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_disable_interrupts)
+
+/*
+ * pmu layout:
+ * 0x0000: PMU TAG (RO) (0xCAFECAFE)
+ * 0x0004: PMU VERSION ID (RO) (0x00000000)
+ * 0x0008: CLOCK ENABLE (RW) (31:1 SBZ, 0 CLOCK STATE)
+ */
+void MOCKABLE(kbase_pm_clock_on)(kbase_device *kbdev)
+{
+ OSK_ASSERT( NULL != kbdev );
+
+ if (kbdev->pm.gpu_powered)
+ {
+ /* Already turned on */
+ return;
+ }
+
+ KBASE_TRACE_ADD( kbdev, PM_GPU_ON, NULL, NULL, 0u, 0u );
+
+ /* The GPU is going to transition, so unset the wait queues until the policy
+ * informs us that the transition is complete */
+ osk_waitq_clear(&kbdev->pm.power_up_waitqueue);
+ osk_waitq_clear(&kbdev->pm.power_down_waitqueue);
+
+ if (kbase_device_has_feature(kbdev, KBASE_FEATURE_HAS_MODEL_PMU))
+ kbase_os_reg_write(kbdev, 0x4008, 1);
+
+ if (kbdev->pm.callback_power_on && kbdev->pm.callback_power_on(kbdev))
+ {
+ /* GPU state was lost, reset GPU to ensure it is in a consistent state */
+ kbase_pm_init_hw(kbdev);
+ }
+
+ osk_spinlock_irq_lock(&kbdev->pm.gpu_powered_lock);
+ kbdev->pm.gpu_powered = MALI_TRUE;
+ osk_spinlock_irq_unlock(&kbdev->pm.gpu_powered_lock);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_clock_on)
+
+void MOCKABLE(kbase_pm_clock_off)(kbase_device *kbdev)
+{
+ OSK_ASSERT( NULL != kbdev );
+
+ if (!kbdev->pm.gpu_powered)
+ {
+ /* Already turned off */
+ return;
+ }
+
+ KBASE_TRACE_ADD( kbdev, PM_GPU_OFF, NULL, NULL, 0u, 0u );
+
+ /* Ensure that any IRQ handlers have finished */
+ kbase_synchronize_irqs(kbdev);
+
+ /* The GPU power may be turned off from this point */
+ osk_spinlock_irq_lock(&kbdev->pm.gpu_powered_lock);
+ kbdev->pm.gpu_powered = MALI_FALSE;
+ osk_spinlock_irq_unlock(&kbdev->pm.gpu_powered_lock);
+
+ if (kbdev->pm.callback_power_off)
+ {
+ kbdev->pm.callback_power_off(kbdev);
+ }
+
+ if (kbase_device_has_feature(kbdev, KBASE_FEATURE_HAS_MODEL_PMU))
+ kbase_os_reg_write(kbdev, 0x4008, 0);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_clock_off)
+
+struct kbasep_reset_timeout_data
+{
+ mali_bool timed_out;
+ kbase_device *kbdev;
+};
+
+static void kbasep_reset_timeout(void *data)
+{
+ struct kbasep_reset_timeout_data *rtdata = (struct kbasep_reset_timeout_data*)data;
+
+ rtdata->timed_out = 1;
+
+ /* Set the wait queue to wake up kbase_pm_init_hw even though the reset hasn't completed */
+ kbase_pm_reset_done(rtdata->kbdev);
+}
+
+static void kbase_pm_hw_issues(kbase_device *kbdev)
+{
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8443))
+ {
+ /* Needed due to MIDBASE-1494: LS_PAUSEBUFFER_DISABLE. See PRLAM-8443. */
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(SHADER_CONFIG), 0x00010000, NULL);
+ }
+}
+
+mali_error kbase_pm_init_hw(kbase_device *kbdev)
+{
+ osk_timer timer;
+ struct kbasep_reset_timeout_data rtdata;
+ osk_error osk_err;
+
+ OSK_ASSERT( NULL != kbdev );
+
+ /* Ensure the clock is on before attempting to access the hardware */
+ if (!kbdev->pm.gpu_powered)
+ {
+ /* The GPU is going to transition, so unset the wait queues until the policy
+ * informs us that the transition is complete */
+ osk_waitq_clear(&kbdev->pm.power_up_waitqueue);
+ osk_waitq_clear(&kbdev->pm.power_down_waitqueue);
+
+ if (kbase_device_has_feature(kbdev, KBASE_FEATURE_HAS_MODEL_PMU))
+ kbase_os_reg_write(kbdev, 0x4008, 1);
+
+ if (kbdev->pm.callback_power_on)
+ kbdev->pm.callback_power_on(kbdev);
+
+ osk_spinlock_irq_lock(&kbdev->pm.gpu_powered_lock);
+ kbdev->pm.gpu_powered = MALI_TRUE;
+ osk_spinlock_irq_unlock(&kbdev->pm.gpu_powered_lock);
+ }
+
+ /* Ensure interrupts are off to begin with, this also clears any outstanding interrupts */
+ kbase_pm_disable_interrupts(kbdev);
+
+ /* Soft reset the GPU */
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_COMMAND), 1, NULL);
+
+ /* Unmask the reset complete interrupt only */
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_IRQ_MASK), (1<<8), NULL);
+
+ /* If the GPU never asserts the reset interrupt we just assume that the reset has completed */
+ if (kbase_device_has_feature(kbdev, KBASE_FEATURE_LACKS_RESET_INT))
+ {
+ goto out;
+ }
+
+ /* Initialize a structure for tracking the status of the reset */
+ rtdata.kbdev = kbdev;
+ rtdata.timed_out = 0;
+
+ /* Create a timer to use as a timeout on the reset */
+ osk_err = osk_timer_on_stack_init(&timer);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ osk_timer_callback_set(&timer, kbasep_reset_timeout, &rtdata);
+ osk_err = osk_timer_start(&timer, RESET_TIMEOUT);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ osk_timer_on_stack_term(&timer);
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+ /* Wait for the RESET_COMPLETED interrupt to be raised,
+ * we use the "power up" waitqueue since it isn't in use yet */
+ osk_waitq_wait(&kbdev->pm.power_up_waitqueue);
+
+ if (rtdata.timed_out == 0)
+ {
+ /* GPU has been reset */
+ osk_timer_stop(&timer);
+ osk_timer_on_stack_term(&timer);
+
+ goto out;
+ }
+
+ /* No interrupt has been received - check if the RAWSTAT register says the reset has completed */
+ if (kbase_reg_read(kbdev, GPU_CONTROL_REG(GPU_IRQ_RAWSTAT), NULL) & (1<<8))
+ {
+ /* The interrupt is set in the RAWSTAT; this suggests that the interrupts are not getting to the CPU */
+ OSK_PRINT_WARN(OSK_BASE_PM, "Reset interrupt didn't reach CPU. Check interrupt assignments.\n");
+ /* If interrupts aren't working we can't continue. */
+ osk_timer_on_stack_term(&timer);
+ goto out;
+ }
+
+ /* The GPU doesn't seem to be responding to the reset so try a hard reset */
+ OSK_PRINT_WARN(OSK_BASE_PM, "Failed to soft reset GPU, attempting a hard reset\n");
+ kbase_reg_write(kbdev, GPU_CONTROL_REG(GPU_COMMAND), 2, NULL);
+
+ /* Restart the timer to wait for the hard reset to complete */
+ rtdata.timed_out = 0;
+ osk_err = osk_timer_start(&timer, RESET_TIMEOUT);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ osk_timer_on_stack_term(&timer);
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ /* Wait for the RESET_COMPLETED interrupt to be raised,
+ * we use the "power up" waitqueue since it isn't in use yet */
+ osk_waitq_wait(&kbdev->pm.power_up_waitqueue);
+
+ if (rtdata.timed_out == 0)
+ {
+ /* GPU has been reset */
+ osk_timer_stop(&timer);
+ osk_timer_on_stack_term(&timer);
+
+ goto out;
+ }
+
+ osk_timer_on_stack_term(&timer);
+
+ OSK_PRINT_ERROR(OSK_BASE_PM, "Failed to reset the GPU\n");
+
+ /* The GPU still hasn't reset, give up */
+ return MALI_ERROR_FUNCTION_FAILED;
+
+out:
+ /* If cycle counter was in use-re enable it */
+ osk_spinlock_irq_lock( &kbdev->pm.gpu_cycle_counter_requests_lock );
+
+ if ( kbdev->pm.gpu_cycle_counter_requests )
+ {
+ kbase_reg_write( kbdev, GPU_CONTROL_REG(GPU_COMMAND), GPU_COMMAND_CYCLE_COUNT_START, NULL );
+ }
+
+ osk_spinlock_irq_unlock( &kbdev->pm.gpu_cycle_counter_requests_lock );
+
+ kbase_pm_hw_issues(kbdev);
+
+ return MALI_ERROR_NONE;
+}
+KBASE_EXPORT_TEST_API(kbase_pm_init_hw)
+
+void kbase_pm_request_gpu_cycle_counter( kbase_device *kbdev )
+{
+ OSK_ASSERT( kbdev != NULL );
+
+ OSK_ASSERT( kbdev->pm.gpu_powered );
+
+ osk_spinlock_irq_lock( &kbdev->pm.gpu_cycle_counter_requests_lock );
+
+ OSK_ASSERT( kbdev->pm.gpu_cycle_counter_requests < INT_MAX );
+
+ ++kbdev->pm.gpu_cycle_counter_requests;
+
+ if ( 1 == kbdev->pm.gpu_cycle_counter_requests )
+ {
+ kbase_reg_write( kbdev, GPU_CONTROL_REG(GPU_COMMAND), GPU_COMMAND_CYCLE_COUNT_START, NULL );
+ }
+ osk_spinlock_irq_unlock( &kbdev->pm.gpu_cycle_counter_requests_lock );
+}
+KBASE_EXPORT_TEST_API(kbase_pm_request_gpu_cycle_counter)
+
+void kbase_pm_release_gpu_cycle_counter( kbase_device *kbdev )
+{
+ OSK_ASSERT( kbdev != NULL );
+
+ osk_spinlock_irq_lock( &kbdev->pm.gpu_cycle_counter_requests_lock );
+
+ OSK_ASSERT( kbdev->pm.gpu_cycle_counter_requests > 0 );
+
+ --kbdev->pm.gpu_cycle_counter_requests;
+
+ if ( 0 == kbdev->pm.gpu_cycle_counter_requests )
+ {
+ kbase_reg_write( kbdev, GPU_CONTROL_REG(GPU_COMMAND), GPU_COMMAND_CYCLE_COUNT_STOP, NULL );
+ }
+ osk_spinlock_irq_unlock( &kbdev->pm.gpu_cycle_counter_requests_lock );
+}
+KBASE_EXPORT_TEST_API(kbase_pm_release_gpu_cycle_counter)
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_pm_metrics.c
+ * Metrics for power management
+ */
+
+#include <osk/mali_osk.h>
+
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_kbase_pm.h>
+
+/* When VSync is being hit aim for utilisation between 70-90% */
+#define KBASE_PM_VSYNC_MIN_UTILISATION 70
+#define KBASE_PM_VSYNC_MAX_UTILISATION 90
+/* Otherwise aim for 10-40% */
+#define KBASE_PM_NO_VSYNC_MIN_UTILISATION 10
+#define KBASE_PM_NO_VSYNC_MAX_UTILISATION 40
+
+/* Frequency that DVFS clock frequency decisions should be made */
+#define KBASE_PM_DVFS_FREQUENCY 500
+
+static void dvfs_callback(void *data)
+{
+ kbase_device *kbdev;
+ kbase_pm_dvfs_action action;
+ osk_error ret;
+
+ OSK_ASSERT(data != NULL);
+
+ kbdev = (kbase_device*)data;
+ action = kbase_pm_get_dvfs_action(kbdev);
+
+ switch(action) {
+ case KBASE_PM_DVFS_NOP:
+ break;
+ case KBASE_PM_DVFS_CLOCK_UP:
+ /* Do whatever is required to increase the clock frequency */
+ break;
+ case KBASE_PM_DVFS_CLOCK_DOWN:
+ /* Do whatever is required to decrease the clock frequency */
+ break;
+ }
+
+ osk_spinlock_irq_lock(&kbdev->pm.metrics.lock);
+ if (kbdev->pm.metrics.timer_active)
+ {
+ ret = osk_timer_start(&kbdev->pm.metrics.timer, KBASE_PM_DVFS_FREQUENCY);
+ if (ret != OSK_ERR_NONE)
+ {
+ /* Handle the situation where the timer cannot be restarted */
+ }
+ }
+ osk_spinlock_irq_unlock(&kbdev->pm.metrics.lock);
+}
+
+mali_error kbasep_pm_metrics_init(kbase_device *kbdev)
+{
+ osk_error osk_err;
+ mali_error ret;
+
+ OSK_ASSERT(kbdev != NULL);
+
+ kbdev->pm.metrics.vsync_hit = 0;
+ kbdev->pm.metrics.utilisation = 0;
+
+ kbdev->pm.metrics.time_period_start = osk_time_now();
+ kbdev->pm.metrics.time_busy = 0;
+ kbdev->pm.metrics.time_idle = 0;
+ kbdev->pm.metrics.gpu_active = MALI_TRUE;
+ kbdev->pm.metrics.timer_active = MALI_TRUE;
+
+ osk_err = osk_spinlock_irq_init(&kbdev->pm.metrics.lock, OSK_LOCK_ORDER_PM_METRICS);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ ret = MALI_ERROR_FUNCTION_FAILED;
+ goto out;
+ }
+
+ osk_err = osk_timer_init(&kbdev->pm.metrics.timer);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ ret = MALI_ERROR_FUNCTION_FAILED;
+ goto spinlock_free;
+ }
+ osk_timer_callback_set(&kbdev->pm.metrics.timer, dvfs_callback, kbdev);
+ osk_err = osk_timer_start(&kbdev->pm.metrics.timer, KBASE_PM_DVFS_FREQUENCY);
+ if (OSK_ERR_NONE != osk_err)
+ {
+ ret = MALI_ERROR_FUNCTION_FAILED;
+ goto timer_free;
+ }
+
+ kbase_pm_register_vsync_callback(kbdev);
+ ret = MALI_ERROR_NONE;
+ goto out;
+
+timer_free:
+ osk_timer_stop(&kbdev->pm.metrics.timer);
+ osk_timer_term(&kbdev->pm.metrics.timer);
+spinlock_free:
+ osk_spinlock_irq_term(&kbdev->pm.metrics.lock);
+out:
+ return ret;
+}
+KBASE_EXPORT_TEST_API(kbasep_pm_metrics_init)
+
+void kbasep_pm_metrics_term(kbase_device *kbdev)
+{
+ OSK_ASSERT(kbdev != NULL);
+
+ osk_spinlock_irq_lock(&kbdev->pm.metrics.lock);
+ kbdev->pm.metrics.timer_active = MALI_FALSE;
+ osk_spinlock_irq_unlock(&kbdev->pm.metrics.lock);
+
+ osk_timer_stop(&kbdev->pm.metrics.timer);
+ osk_timer_term(&kbdev->pm.metrics.timer);
+
+ kbase_pm_unregister_vsync_callback(kbdev);
+
+ osk_spinlock_irq_term(&kbdev->pm.metrics.lock);
+}
+KBASE_EXPORT_TEST_API(kbasep_pm_metrics_term)
+
+void kbasep_pm_record_gpu_idle(kbase_device *kbdev)
+{
+ osk_ticks now = osk_time_now();
+
+ OSK_ASSERT(kbdev != NULL);
+
+ osk_spinlock_irq_lock(&kbdev->pm.metrics.lock);
+
+ OSK_ASSERT(kbdev->pm.metrics.gpu_active == MALI_TRUE);
+
+ kbdev->pm.metrics.gpu_active = MALI_FALSE;
+
+ kbdev->pm.metrics.time_busy += osk_time_elapsed(kbdev->pm.metrics.time_period_start, now);
+ kbdev->pm.metrics.time_period_start = now;
+
+ osk_spinlock_irq_unlock(&kbdev->pm.metrics.lock);
+}
+KBASE_EXPORT_TEST_API(kbasep_pm_record_gpu_idle)
+
+void kbasep_pm_record_gpu_active(kbase_device *kbdev)
+{
+ osk_ticks now = osk_time_now();
+
+ OSK_ASSERT(kbdev != NULL);
+
+ osk_spinlock_irq_lock(&kbdev->pm.metrics.lock);
+
+ OSK_ASSERT(kbdev->pm.metrics.gpu_active == MALI_FALSE);
+
+ kbdev->pm.metrics.gpu_active = MALI_TRUE;
+
+ kbdev->pm.metrics.time_idle += osk_time_elapsed(kbdev->pm.metrics.time_period_start, now);
+ kbdev->pm.metrics.time_period_start = now;
+
+ osk_spinlock_irq_unlock(&kbdev->pm.metrics.lock);
+}
+KBASE_EXPORT_TEST_API(kbasep_pm_record_gpu_active)
+
+void kbase_pm_report_vsync(kbase_device *kbdev, int buffer_updated)
+{
+ OSK_ASSERT(kbdev != NULL);
+
+ osk_spinlock_irq_lock(&kbdev->pm.metrics.lock);
+ kbdev->pm.metrics.vsync_hit = buffer_updated;
+ osk_spinlock_irq_unlock(&kbdev->pm.metrics.lock);
+}
+KBASE_EXPORT_TEST_API(kbase_pm_report_vsync)
+
+kbase_pm_dvfs_action kbase_pm_get_dvfs_action(kbase_device *kbdev)
+{
+ int utilisation;
+ kbase_pm_dvfs_action action;
+ osk_ticks now = osk_time_now();
+
+ OSK_ASSERT(kbdev != NULL);
+
+ osk_spinlock_irq_lock(&kbdev->pm.metrics.lock);
+
+ if (kbdev->pm.metrics.gpu_active)
+ {
+ kbdev->pm.metrics.time_busy += osk_time_elapsed(kbdev->pm.metrics.time_period_start, now);
+ kbdev->pm.metrics.time_period_start = now;
+ }
+ else
+ {
+ kbdev->pm.metrics.time_idle += osk_time_elapsed(kbdev->pm.metrics.time_period_start, now);
+ kbdev->pm.metrics.time_period_start = now;
+ }
+
+ if (kbdev->pm.metrics.time_idle + kbdev->pm.metrics.time_busy == 0)
+ {
+ /* No data - so we return NOP */
+ action = KBASE_PM_DVFS_NOP;
+ goto out;
+ }
+
+ utilisation = (100*kbdev->pm.metrics.time_busy) / (kbdev->pm.metrics.time_idle + kbdev->pm.metrics.time_busy);
+
+ if (kbdev->pm.metrics.vsync_hit)
+ {
+ /* VSync is being met */
+ if (utilisation < KBASE_PM_VSYNC_MIN_UTILISATION)
+ {
+ action = KBASE_PM_DVFS_CLOCK_DOWN;
+ }
+ else if (utilisation > KBASE_PM_VSYNC_MAX_UTILISATION)
+ {
+ action = KBASE_PM_DVFS_CLOCK_UP;
+ }
+ else
+ {
+ action = KBASE_PM_DVFS_NOP;
+ }
+ }
+ else
+ {
+ /* VSync is being missed */
+ if (utilisation < KBASE_PM_NO_VSYNC_MIN_UTILISATION)
+ {
+ action = KBASE_PM_DVFS_CLOCK_DOWN;
+ }
+ else if (utilisation > KBASE_PM_NO_VSYNC_MAX_UTILISATION)
+ {
+ action = KBASE_PM_DVFS_CLOCK_UP;
+ }
+ else
+ {
+ action = KBASE_PM_DVFS_NOP;
+ }
+ }
+
+out:
+
+ kbdev->pm.metrics.time_idle = 0;
+ kbdev->pm.metrics.time_busy = 0;
+
+ osk_spinlock_irq_unlock(&kbdev->pm.metrics.lock);
+
+ return action;
+}
+KBASE_EXPORT_TEST_API(kbase_pm_get_dvfs_action)
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_pm_metrics_dummy.c
+ * Dummy Metrics for power management.
+ */
+
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_kbase_pm.h>
+
+void kbase_pm_register_vsync_callback(kbase_device *kbdev)
+{
+ OSK_ASSERT(kbdev != NULL);
+
+ /* no VSync metrics will be available */
+ kbdev->pm.metrics.platform_data = NULL;
+}
+
+void kbase_pm_unregister_vsync_callback(kbase_device *kbdev)
+{
+ OSK_ASSERT(kbdev != NULL);
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_security.c
+ * Base kernel security capability API
+ */
+
+#include <osk/mali_osk.h>
+#include <kbase/src/common/mali_kbase.h>
+
+/**
+ * kbase_security_has_capability - see mali_kbase_caps.h for description.
+ */
+
+mali_bool kbase_security_has_capability(kbase_context *kctx, kbase_security_capability cap, u32 flags)
+{
+ /* Assume failure */
+ mali_bool access_allowed = MALI_FALSE;
+ mali_bool audit = (KBASE_SEC_FLAG_AUDIT & flags)? MALI_TRUE : MALI_FALSE;
+
+ OSK_ASSERT(NULL != kctx);
+ CSTD_UNUSED(kctx);
+
+ /* Detect unsupported flags */
+ OSK_ASSERT(((~KBASE_SEC_FLAG_MASK) & flags) == 0);
+
+ /* Determine if access is allowed for the given cap */
+ switch(cap)
+ {
+ case KBASE_SEC_MODIFY_PRIORITY:
+#if KBASE_HWCNT_DUMP_BYPASS_ROOT
+ access_allowed = TRUE;
+#else
+ if (osk_is_privileged() == MALI_TRUE)
+ {
+ access_allowed = TRUE;
+ }
+#endif
+ break;
+ case KBASE_SEC_INSTR_HW_COUNTERS_COLLECT:
+ /* Access is granted only if the caller is privileged */
+#if KBASE_HWCNT_DUMP_BYPASS_ROOT
+ access_allowed = TRUE;
+#else
+ if (osk_is_privileged() == MALI_TRUE)
+ {
+ access_allowed = TRUE;
+ }
+#endif
+ break;
+ }
+
+ /* Report problem if requested */
+ if(MALI_FALSE == access_allowed)
+ {
+ if(MALI_FALSE != audit)
+ {
+ OSK_PRINT_WARN(OSK_BASE_CORE, "Security capability failure: %d, %p", cap, (void *)kctx);
+ }
+ }
+
+ return access_allowed;
+}
+KBASE_EXPORT_TEST_API(kbase_security_has_capability)
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_security.h
+ * Base kernel security capability APIs
+ */
+
+#ifndef _KBASE_SECURITY_H_
+#define _KBASE_SECURITY_H_
+
+/* Security flags */
+#define KBASE_SEC_FLAG_NOAUDIT (0u << 0) /* Silently handle privilege failure */
+#define KBASE_SEC_FLAG_AUDIT (1u << 0) /* Write audit message on privilege failure */
+#define KBASE_SEC_FLAG_MASK (KBASE_SEC_FLAG_AUDIT) /* Mask of all valid flag bits */
+
+/* List of unique capabilities that have security access privileges */
+typedef enum {
+ /* Instrumentation Counters access privilege */
+ KBASE_SEC_INSTR_HW_COUNTERS_COLLECT = 1,
+ KBASE_SEC_MODIFY_PRIORITY
+ /* Add additional access privileges here */
+} kbase_security_capability;
+
+
+/**
+ * kbase_security_has_capability - determine whether a task has a particular effective capability
+ * @param[in] kctx The task context.
+ * @param[in] cap The capability to check for.
+ * @param[in] flags Additional configuration information
+ * Such as whether to write an audit message or not.
+ * @return MALI_TRUE if success (capability is allowed), MALI_FALSE otherwise.
+ */
+
+mali_bool kbase_security_has_capability(kbase_context *kctx, kbase_security_capability cap, u32 flags);
+
+#endif /* _KBASE_SECURITY_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#include <kbase/src/common/mali_kbase.h>
+
+/**
+ * @file mali_kbase_softjobs.c
+ *
+ * This file implements the logic behind software only jobs that are
+ * executed within the driver rather than being handed over to the GPU.
+ */
+
+static base_jd_event_code kbase_dump_cpu_gpu_time(kbase_context *kctx, kbase_jd_atom *katom)
+{
+ kbase_va_region *reg;
+ osk_phy_addr addr;
+ u64 pfn;
+ u32 offset;
+ char *page;
+ osk_timeval tv;
+ base_dump_cpu_gpu_counters data;
+ u64 system_time;
+ u64 cycle_counter;
+ mali_addr64 jc = katom->jc;
+
+ u32 hi1, hi2;
+
+ OSK_MEMSET(&data, 0, sizeof(data));
+
+ /* GPU needs to be powered to read the cycle counters, the jctx->lock protects this check */
+ if (!katom->bag->has_pm_ctx_reference)
+ {
+ kbase_pm_context_active(kctx->kbdev);
+ katom->bag->has_pm_ctx_reference = MALI_TRUE;
+ }
+
+ /* Read hi, lo, hi to ensure that overflow from lo to hi is handled correctly */
+ do {
+ hi1 = kbase_reg_read(kctx->kbdev, GPU_CONTROL_REG(CYCLE_COUNT_HI), NULL);
+ cycle_counter = kbase_reg_read(kctx->kbdev, GPU_CONTROL_REG(CYCLE_COUNT_LO), NULL);
+ hi2 = kbase_reg_read(kctx->kbdev, GPU_CONTROL_REG(CYCLE_COUNT_HI), NULL);
+ cycle_counter |= (((u64)hi1) << 32);
+ } while (hi1 != hi2);
+
+ /* Read hi, lo, hi to ensure that overflow from lo to hi is handled correctly */
+ do {
+ hi1 = kbase_reg_read(kctx->kbdev, GPU_CONTROL_REG(TIMESTAMP_HI), NULL);
+ system_time = kbase_reg_read(kctx->kbdev, GPU_CONTROL_REG(TIMESTAMP_LO), NULL);
+ hi2 = kbase_reg_read(kctx->kbdev, GPU_CONTROL_REG(TIMESTAMP_HI), NULL);
+ system_time |= (((u64)hi1) << 32);
+ } while (hi1 != hi2);
+
+ /* Record the CPU's idea of current time */
+ osk_gettimeofday(&tv);
+
+ data.sec = tv.tv_sec;
+ data.usec = tv.tv_usec;
+ data.system_time = system_time;
+ data.cycle_counter = cycle_counter;
+
+ pfn = jc >> 12;
+ offset = jc & 0xFFF;
+
+ if (offset > 0x1000-sizeof(data))
+ {
+ /* Wouldn't fit in the page */
+ return BASE_JD_EVENT_JOB_CANCELLED;
+ }
+
+ reg = kbase_region_lookup(kctx, jc);
+ if (!reg)
+ {
+ return BASE_JD_EVENT_JOB_CANCELLED;
+ }
+
+ if (! (reg->flags & KBASE_REG_GPU_WR) )
+ {
+ /* Region is not writable by GPU so we won't write to it either */
+ return BASE_JD_EVENT_JOB_CANCELLED;
+ }
+
+ if (!reg->phy_pages)
+ {
+ return BASE_JD_EVENT_JOB_CANCELLED;
+ }
+
+ addr = reg->phy_pages[pfn - reg->start_pfn];
+ if (!addr)
+ {
+ return BASE_JD_EVENT_JOB_CANCELLED;
+ }
+
+ page = osk_kmap(addr);
+ if (!page)
+ {
+ return BASE_JD_EVENT_JOB_CANCELLED;
+ }
+ memcpy(page+offset, &data, sizeof(data));
+ osk_kunmap(addr, page);
+
+ return BASE_JD_EVENT_DONE;
+}
+
+void kbase_process_soft_job( kbase_context *kctx, kbase_jd_atom *katom )
+{
+ switch(katom->core_req) {
+ case BASE_JD_REQ_SOFT_DUMP_CPU_GPU_TIME:
+ katom->event.event_code = kbase_dump_cpu_gpu_time( kctx, katom);
+ break;
+ }
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+
+/* ***** IMPORTANT: THIS IS NOT A NORMAL HEADER FILE *****
+ * ***** DO NOT INCLUDE DIRECTLY *****
+ * ***** THE LACK OF HEADER GUARDS IS INTENTIONAL ***** */
+
+/*
+ * The purpose of this header file is just to contain a list of trace code idenitifers
+ *
+ * Each identifier is wrapped in a macro, so that its string form and enum form can be created
+ *
+ * Each macro is separated with a comma, to allow insertion into an array initializer or enum definition block.
+ *
+ * This allows automatic creation of an enum and a corresponding array of strings
+ *
+ * Before #including, the includer MUST #define KBASE_TRACE_CODE_MAKE_CODE.
+ * After #including, the includer MUST #under KBASE_TRACE_CODE_MAKE_CODE.
+ *
+ * e.g.:
+ * #define KBASE_TRACE_CODE( X ) KBASE_TRACE_CODE_ ## X
+ * typedef enum
+ * {
+ * #define KBASE_TRACE_CODE_MAKE_CODE( X ) KBASE_TRACE_CODE( X )
+ * #include "mali_kbase_trace_defs.h"
+ * #undef KBASE_TRACE_CODE_MAKE_CODE
+ * } kbase_trace_code;
+ *
+ * IMPORTANT: THIS FILE MUST NOT BE USED FOR ANY OTHER PURPOSE OTHER THAN THE ABOVE
+ *
+ *
+ * The use of the macro here is:
+ * - KBASE_TRACE_CODE_MAKE_CODE( X )
+ *
+ * Which produces:
+ * - For an enum, KBASE_TRACE_CODE_X
+ * - For a string, "X"
+ *
+ *
+ * For example:
+ * - KBASE_TRACE_CODE_MAKE_CODE( JM_JOB_COMPLETE ) expands to:
+ * - KBASE_TRACE_CODE_JM_JOB_COMPLETE for the enum
+ * - "JM_JOB_COMPLETE" for the string
+ * - To use it to trace an event, do:
+ * - KBASE_TRACE_ADD( kbdev, JM_JOB_COMPLETE, subcode, kctx, uatom, val );
+ */
+
+/*
+ * Core events
+ */
+KBASE_TRACE_CODE_MAKE_CODE( CORE_CTX_DESTROY ), /* no info_val, no gpu_addr, no atom */
+KBASE_TRACE_CODE_MAKE_CODE( CORE_CTX_HWINSTR_TERM ), /* no info_val, no gpu_addr, no atom */
+
+/*
+ * Job Slot management events
+ */
+
+KBASE_TRACE_CODE_MAKE_CODE( JM_IRQ ), /* info_val==irq rawstat at start */
+KBASE_TRACE_CODE_MAKE_CODE( JM_IRQ_END ), /* info_val==jobs processed */
+/* In the following:
+ *
+ * - ctx is set if a corresponding job found (NULL otherwise, e.g. some soft-stop cases)
+ * - uatom==kernel-side mapped uatom address (for correlation with user-side)
+ */
+KBASE_TRACE_CODE_MAKE_CODE( JM_JOB_DONE ), /* info_val==exit code; gpu_addr==chain gpuaddr */
+KBASE_TRACE_CODE_MAKE_CODE( JM_SUBMIT ), /* gpu_addr==JSn_HEAD_NEXT written, info_val==lower 32 bits of affinity */
+
+/* gpu_addr is as follows:
+ * - If JSn_STATUS active after soft-stop, val==gpu addr written to JSn_HEAD on submit
+ * - otherwise gpu_addr==0 */
+KBASE_TRACE_CODE_MAKE_CODE( JM_SOFTSTOP ),
+KBASE_TRACE_CODE_MAKE_CODE( JM_HARDSTOP ), /* gpu_addr==JSn_HEAD read */
+
+KBASE_TRACE_CODE_MAKE_CODE( JM_UPDATE_HEAD ), /* gpu_addr==JSn_TAIL read */
+/* gpu_addr is as follows:
+ * - If JSn_STATUS active before soft-stop, val==JSn_HEAD
+ * - otherwise gpu_addr==0 */
+KBASE_TRACE_CODE_MAKE_CODE( JM_CHECK_HEAD ), /* gpu_addr==JSn_HEAD read */
+
+KBASE_TRACE_CODE_MAKE_CODE( JM_FLUSH_WORKQS ),
+KBASE_TRACE_CODE_MAKE_CODE( JM_FLUSH_WORKQS_DONE ),
+
+KBASE_TRACE_CODE_MAKE_CODE( JM_ZAP_NON_SCHEDULED ), /* info_val == is_scheduled */
+KBASE_TRACE_CODE_MAKE_CODE( JM_ZAP_SCHEDULED ), /* info_val == is_scheduled */
+KBASE_TRACE_CODE_MAKE_CODE( JM_ZAP_DONE ),
+
+KBASE_TRACE_CODE_MAKE_CODE( JM_SLOT_SOFT_OR_HARD_STOP ), /* info_val == nr jobs submitted */
+KBASE_TRACE_CODE_MAKE_CODE( JM_SLOT_EVICT ), /* gpu_addr==JSn_HEAD_NEXT last written */
+KBASE_TRACE_CODE_MAKE_CODE( JM_SUBMIT_AFTER_RESET ),
+
+
+/*
+ * Job dispatch events
+ */
+KBASE_TRACE_CODE_MAKE_CODE( JD_DONE ), /* gpu_addr==value to write into JSn_HEAD */
+KBASE_TRACE_CODE_MAKE_CODE( JD_DONE_WORKER ), /* gpu_addr==value to write into JSn_HEAD */
+KBASE_TRACE_CODE_MAKE_CODE( JD_DONE_WORKER_END ), /* gpu_addr==value to write into JSn_HEAD */
+KBASE_TRACE_CODE_MAKE_CODE( JD_DONE_TRY_RUN_NEXT_JOB ), /* gpu_addr==value to write into JSn_HEAD */
+
+KBASE_TRACE_CODE_MAKE_CODE( JD_ZAP_CONTEXT ), /* gpu_addr==0, info_val==0, uatom==0 */
+KBASE_TRACE_CODE_MAKE_CODE( JD_CANCEL ), /* gpu_addr==value to write into JSn_HEAD */
+KBASE_TRACE_CODE_MAKE_CODE( JD_CANCEL_WORKER ), /* gpu_addr==value to write into JSn_HEAD */
+
+/*
+ * Scheduler Core events
+ */
+
+KBASE_TRACE_CODE_MAKE_CODE( JS_RETAIN_CTX_NOLOCK ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_ADD_JOB ), /* gpu_addr==value to write into JSn_HEAD */
+KBASE_TRACE_CODE_MAKE_CODE( JS_REMOVE_JOB ), /* gpu_addr==last value written/would be written to JSn_HEAD */
+KBASE_TRACE_CODE_MAKE_CODE( JS_RETAIN_CTX ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_RELEASE_CTX ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_TRY_SCHEDULE_HEAD_CTX ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_JOB_DONE_TRY_RUN_NEXT_JOB ), /* gpu_addr==value to write into JSn_HEAD */
+KBASE_TRACE_CODE_MAKE_CODE( JS_JOB_DONE_RETRY_NEEDED ), /* gpu_addr==value to write into JSn_HEAD */
+KBASE_TRACE_CODE_MAKE_CODE( JS_FAST_START_EVICTS_CTX ), /* kctx is the one being evicted, info_val == kctx to put in */
+KBASE_TRACE_CODE_MAKE_CODE( JS_AFFINITY_SUBMIT_TO_BLOCKED ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_AFFINITY_CURRENT ), /* info_val == lower 32 bits of affinity */
+KBASE_TRACE_CODE_MAKE_CODE( JS_CORE_REF_REQUEST_CORES_FAILED ), /* info_val == lower 32 bits of affinity */
+KBASE_TRACE_CODE_MAKE_CODE( JS_CORE_REF_REGISTER_INUSE_FAILED ), /* info_val == lower 32 bits of affinity */
+KBASE_TRACE_CODE_MAKE_CODE( JS_CORE_REF_REQUEST_ON_RECHECK_FAILED ), /* info_val == lower 32 bits of rechecked affinity */
+KBASE_TRACE_CODE_MAKE_CODE( JS_CORE_REF_REGISTER_ON_RECHECK_FAILED ), /* info_val == lower 32 bits of rechecked affinity */
+KBASE_TRACE_CODE_MAKE_CODE( JS_CORE_REF_AFFINITY_WOULD_VIOLATE ), /* info_val == lower 32 bits of affinity */
+KBASE_TRACE_CODE_MAKE_CODE( JS_CTX_ATTR_NOW_ON_CTX ), /* info_val == the ctx attribute now on ctx */
+KBASE_TRACE_CODE_MAKE_CODE( JS_CTX_ATTR_NOW_ON_RUNPOOL ), /* info_val == the ctx attribute now on runpool */
+KBASE_TRACE_CODE_MAKE_CODE( JS_CTX_ATTR_NOW_OFF_CTX ), /* info_val == the ctx attribute now off ctx */
+KBASE_TRACE_CODE_MAKE_CODE( JS_CTX_ATTR_NOW_OFF_RUNPOOL ), /* info_val == the ctx attribute now off runpool */
+
+
+/*
+ * Scheduler Policy events
+ */
+
+KBASE_TRACE_CODE_MAKE_CODE( JS_POLICY_INIT_CTX ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_POLICY_TERM_CTX ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_POLICY_TRY_EVICT_CTX ), /* info_val == whether it was evicted */
+KBASE_TRACE_CODE_MAKE_CODE( JS_POLICY_KILL_ALL_CTX_JOBS ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_POLICY_ENQUEUE_CTX ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_POLICY_DEQUEUE_HEAD_CTX ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_POLICY_RUNPOOL_ADD_CTX ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_POLICY_RUNPOOL_REMOVE_CTX ),
+
+KBASE_TRACE_CODE_MAKE_CODE( JS_POLICY_DEQUEUE_JOB ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_POLICY_DEQUEUE_JOB_IRQ ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_POLICY_ENQUEUE_JOB ), /* gpu_addr==JSn_HEAD to write if the job were run */
+KBASE_TRACE_CODE_MAKE_CODE( JS_POLICY_TIMER_START ),
+KBASE_TRACE_CODE_MAKE_CODE( JS_POLICY_TIMER_END ),
+
+
+/*
+ * Power Management Events
+ */
+KBASE_TRACE_CODE_MAKE_CODE( PM_JOB_SUBMIT_AFTER_POWERING_UP ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_JOB_SUBMIT_AFTER_POWERED_UP ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_PWRON ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_PWROFF ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_CORES_POWERED ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_CORES_CHANGE_DESIRED_ON_POWERUP ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_CORES_CHANGE_DESIRED_ON_POWERDOWN ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_CORES_CHANGE_DESIRED ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_CORES_CHANGE_AVAILABLE ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_REGISTER_CHANGE_SHADER_INUSE ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_REGISTER_CHANGE_SHADER_NEEDED ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_RELEASE_CHANGE_SHADER_INUSE ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_UNREQUEST_CHANGE_SHADER_NEEDED ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_REQUEST_CHANGE_SHADER_NEEDED ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_CONTEXT_ACTIVE ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_CONTEXT_IDLE ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_GPU_ON ),
+KBASE_TRACE_CODE_MAKE_CODE( PM_GPU_OFF ),
+
+
+
+/* Unused code just to make it easier to not have a comma at the end.
+ * All other codes MUST come before this */
+KBASE_TRACE_CODE_MAKE_CODE( DUMMY )
+
+/* ***** THE LACK OF HEADER GUARDS IS INTENTIONAL ***** */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _KBASE_UKU_H_
+#define _KBASE_UKU_H_
+
+#include <uk/mali_uk.h>
+#if MALI_USE_UMP == 1
+#include <ump/ump_common.h>
+#endif /*MALI_USE_UMP == 1*/
+#include <malisw/mali_malisw.h>
+#include <kbase/mali_base_kernel.h>
+#if (MALI_ERROR_INJECT_ON || MALI_NO_MALI)
+#include <kbase/src/common/mali_kbase_model_dummy.h>
+#endif
+
+#include "mali_kbase_gpuprops_types.h"
+
+#define BASE_UK_VERSION_MAJOR 1
+#define BASE_UK_VERSION_MINOR 0
+
+/** 32/64-bit neutral way to represent pointers */
+typedef union kbase_pointer
+{
+ void * value; /**< client should store their pointers here */
+ u32 compat_value; /**< 64-bit kernels should fetch value here when handling 32-bit clients */
+ u64 sizer; /**< Force 64-bit storage for all clients regardless */
+} kbase_pointer;
+
+typedef struct kbase_uk_tmem_alloc
+{
+ uk_header header;
+ /* IN */
+ u32 vsize;
+ u32 psize;
+ u32 extent;
+ u32 flags;
+ mali_bool is_growable;
+ /* OUT */
+ mali_addr64 gpu_addr;
+} kbase_uk_tmem_alloc;
+
+typedef struct kbase_uk_tmem_import
+{
+ uk_header header;
+ /* IN */
+ kbase_pointer phandle;
+ base_tmem_import_type type;
+ u32 padding;
+ /* OUT */
+ mali_addr64 gpu_addr;
+ u64 pages;
+} kbase_uk_tmem_import;
+
+typedef struct kbase_uk_pmem_alloc
+{
+ uk_header header;
+ /* IN */
+ u32 vsize;
+ u32 flags;
+ /* OUT */
+ u16 cookie;
+} kbase_uk_pmem_alloc;
+
+typedef struct kbase_uk_mem_free
+{
+ uk_header header;
+ /* IN */
+ mali_addr64 gpu_addr;
+ /* OUT */
+} kbase_uk_mem_free;
+
+typedef struct kbase_uk_job_submit
+{
+ uk_header header;
+ /* IN */
+ u64 bag_uaddr;
+ u64 core_restriction;
+ u32 offset;
+ u32 size;
+ u32 nr_atoms;
+ /* OUT */
+} kbase_uk_job_submit;
+
+typedef struct kbase_uk_post_term
+{
+ uk_header header;
+} kbase_uk_post_term;
+
+typedef struct kbase_uk_sync_now
+{
+ uk_header header;
+
+ /* IN */
+ base_syncset sset;
+
+ /* OUT */
+} kbase_uk_sync_now;
+
+typedef struct kbase_uk_hwcnt_setup
+{
+ uk_header header;
+
+ /* IN */
+ mali_addr64 dump_buffer;
+ u32 jm_bm;
+ u32 shader_bm;
+ u32 tiler_bm;
+ u32 l3_cache_bm;
+ u32 mmu_l2_bm;
+ /* OUT */
+} kbase_uk_hwcnt_setup;
+
+typedef struct kbase_uk_hwcnt_dump
+{
+ uk_header header;
+} kbase_uk_hwcnt_dump;
+
+typedef struct kbase_uk_hwcnt_clear
+{
+ uk_header header;
+} kbase_uk_hwcnt_clear;
+
+typedef struct kbase_uk_cpuprops
+{
+ uk_header header;
+
+ /* IN */
+ struct base_cpu_props props;
+ /* OUT */
+}kbase_uk_cpuprops;
+
+typedef struct kbase_uk_gpuprops
+{
+ uk_header header;
+
+ /* IN */
+ struct mali_base_gpu_props props;
+ /* OUT */
+}kbase_uk_gpuprops;
+
+typedef struct kbase_uk_tmem_resize
+{
+ uk_header header;
+ /* IN */
+ mali_addr64 gpu_addr;
+ s32 delta;
+ /* OUT */
+ u32 size;
+ base_backing_threshold_status result_subcode;
+} kbase_uk_tmem_resize;
+
+typedef struct kbase_uk_find_cpu_mapping
+{
+ uk_header header;
+ /* IN */
+ mali_addr64 gpu_addr;
+ u64 cpu_addr;
+ u64 size;
+ /* OUT */
+ u64 uaddr;
+ u32 nr_pages;
+ mali_size64 page_off;
+} kbase_uk_find_cpu_mapping;
+
+#define KBASE_GET_VERSION_BUFFER_SIZE 64
+typedef struct kbase_uk_get_ddk_version
+{
+ uk_header header;
+ /* OUT */
+ char version_buffer[KBASE_GET_VERSION_BUFFER_SIZE];
+ u32 version_string_size;
+} kbase_uk_get_ddk_version;
+
+typedef struct kbase_uk_set_flags
+{
+ uk_header header;
+ /* IN */
+ u32 create_flags;
+} kbase_uk_set_flags;
+
+#if MALI_UNIT_TEST
+#define TEST_ADDR_COUNT 4
+#define KBASE_TEST_BUFFER_SIZE 128
+typedef struct kbase_exported_test_data
+{
+ mali_addr64 test_addr[TEST_ADDR_COUNT]; /* memory address */
+ u32 test_addr_pages[TEST_ADDR_COUNT]; /* memory size in pages */
+ struct kbase_context *kctx; /* base context created by process */
+ void *mm; /* pointer to process address space */
+ u8 buffer1[KBASE_TEST_BUFFER_SIZE]; /* unit test defined parameter */
+ u8 buffer2[KBASE_TEST_BUFFER_SIZE]; /* unit test defined parameter */
+} kbase_exported_test_data;
+
+typedef struct kbase_uk_set_test_data
+{
+ uk_header header;
+ /* IN */
+ kbase_exported_test_data test_data;
+} kbase_uk_set_test_data;
+
+#endif /* MALI_UNIT_TEST */
+#if MALI_ERROR_INJECT_ON
+typedef struct kbase_uk_error_params
+{
+ uk_header header;
+ /* IN */
+ kbase_error_params params;
+} kbase_uk_error_params;
+#endif
+
+#if MALI_NO_MALI
+typedef struct kbase_uk_model_control_params
+{
+ uk_header header;
+ /* IN */
+ kbase_model_control_params params;
+} kbase_uk_model_control_params;
+#endif /* MALI_NO_MALI */
+
+
+typedef struct kbase_uk_ext_buff_kds_data
+{
+ uk_header header;
+ kbase_pointer external_resource;
+ int num_res;
+ kbase_pointer file_descriptor;
+} kbase_uk_ext_buff_kds_data;
+
+
+typedef enum kbase_uk_function_id
+{
+ KBASE_FUNC_TMEM_ALLOC = (UK_FUNC_ID + 0),
+ KBASE_FUNC_TMEM_IMPORT,
+ KBASE_FUNC_PMEM_ALLOC,
+ KBASE_FUNC_MEM_FREE,
+
+ KBASE_FUNC_JOB_SUBMIT,
+
+ KBASE_FUNC_SYNC,
+
+ KBASE_FUNC_POST_TERM,
+
+ KBASE_FUNC_HWCNT_SETUP,
+ KBASE_FUNC_HWCNT_DUMP,
+ KBASE_FUNC_HWCNT_CLEAR,
+
+ KBASE_FUNC_CPU_PROPS_REG_DUMP,
+ KBASE_FUNC_GPU_PROPS_REG_DUMP,
+
+ KBASE_FUNC_TMEM_RESIZE,
+
+ KBASE_FUNC_FIND_CPU_MAPPING,
+
+ KBASE_FUNC_GET_VERSION,
+ KBASE_FUNC_EXT_BUFFER_LOCK,
+ KBASE_FUNC_SET_FLAGS
+
+#if MALI_DEBUG
+ , KBASE_FUNC_SET_TEST_DATA
+#endif /* MALI_DEBUG */
+#if MALI_ERROR_INJECT_ON
+ , KBASE_FUNC_INJECT_ERROR
+#endif
+#if MALI_NO_MALI
+ , KBASE_FUNC_MODEL_CONTROL
+#endif /* MALI_NO_MALI */
+
+} kbase_uk_function_id;
+
+#endif /* _KBASE_UKU_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _MIDGARD_REGMAP_H_
+#define _MIDGARD_REGMAP_H_
+
+/*
+ * Begin Register Offsets
+ */
+
+#define GPU_CONTROL_BASE 0x0000
+#define GPU_CONTROL_REG(r) (GPU_CONTROL_BASE + (r))
+#define GPU_ID 0x000 /* (RO) GPU and revision identifier */
+#define L2_FEATURES 0x004 /* (RO) Level 2 cache features */
+#define L3_FEATURES 0x008 /* (RO) Level 3 cache features */
+#define TILER_FEATURES 0x00C /* (RO) Tiler Features */
+#define MEM_FEATURES 0x010 /* (RO) Memory system features */
+#define MMU_FEATURES 0x014 /* (RO) MMU features */
+#define AS_PRESENT 0x018 /* (RO) Address space slots present */
+#define JS_PRESENT 0x01C /* (RO) Job slots present */
+#define GPU_IRQ_RAWSTAT 0x020 /* (RW) */
+#define GPU_IRQ_CLEAR 0x024 /* (WO) */
+#define GPU_IRQ_MASK 0x028 /* (RW) */
+#define GPU_IRQ_STATUS 0x02C /* (RO) */
+
+/* IRQ flags */
+#define GPU_FAULT (1 << 0) /* A GPU Fault has occurred */
+#define MULTIPLE_GPU_FAULTS (1 << 7) /* More than one GPU Fault occurred. */
+#define RESET_COMPLETED (1 << 8) /* Set when a reset has completed. Intended to use with SOFT_RESET
+ commands which may take time.*/
+#define POWER_CHANGED_SINGLE (1 << 9) /* Set when a single core has finished powering up or down. */
+#define POWER_CHANGED_ALL (1 << 10) /* Set when all cores have finished powering up or down
+ and the power manager is idle. */
+
+#define PRFCNT_SAMPLE_COMPLETED (1 << 16) /* Set when a performance count sample has completed. */
+#define CLEAN_CACHES_COMPLETED (1 << 17) /* Set when a cache clean operation has completed. */
+
+#define GPU_IRQ_REG_ALL (GPU_FAULT | MULTIPLE_GPU_FAULTS | RESET_COMPLETED \
+ | POWER_CHANGED_ALL | PRFCNT_SAMPLE_COMPLETED \
+ | CLEAN_CACHES_COMPLETED)
+
+#define GPU_COMMAND 0x030 /* (WO) */
+#define GPU_STATUS 0x034 /* (RO) */
+
+#define GROUPS_L2_COHERENT (1 << 0) /* Cores groups are l2 coherent */
+#define GROUPS_L3_COHERENT (1 << 1) /* Cores groups are l3 coherent */
+
+#define GPU_FAULTSTATUS 0x03C /* (RO) GPU exception type and fault status */
+#define GPU_FAULTADDRESS_LO 0x040 /* (RO) GPU exception fault address, low word */
+#define GPU_FAULTADDRESS_HI 0x044 /* (RO) GPU exception fault address, high word */
+
+#define PWR_KEY 0x050 /* (WO) Power manager key register */
+#define PWR_OVERRIDE0 0x054 /* (RW) Power manager override settings */
+#define PWR_OVERRIDE1 0x058 /* (RW) Power manager override settings */
+
+#define PRFCNT_BASE_LO 0x060 /* (RW) Performance counter memory region base address, low word */
+#define PRFCNT_BASE_HI 0x064 /* (RW) Performance counter memory region base address, high word */
+#define PRFCNT_CONFIG 0x068 /* (RW) Performance counter configuration */
+#define PRFCNT_JM_EN 0x06C /* (RW) Performance counter enable flags for Job Manager */
+#define PRFCNT_SHADER_EN 0x070 /* (RW) Performance counter enable flags for shader cores */
+#define PRFCNT_TILER_EN 0x074 /* (RW) Performance counter enable flags for tiler */
+#define PRFCNT_L3_CACHE_EN 0x078 /* (RW) Performance counter enable flags for L3 cache */
+#define PRFCNT_MMU_L2_EN 0x07C /* (RW) Performance counter enable flags for MMU/L2 cache */
+
+#define CYCLE_COUNT_LO 0x090 /* (RO) Cycle counter, low word */
+#define CYCLE_COUNT_HI 0x094 /* (RO) Cycle counter, high word */
+#define TIMESTAMP_LO 0x098 /* (RO) Global time stamp counter, low word */
+#define TIMESTAMP_HI 0x09C /* (RO) Global time stamp counter, high word */
+
+#define TEXTURE_FEATURES_0 0x0B0 /* (RO) Support flags for indexed texture formats 0..31 */
+#define TEXTURE_FEATURES_1 0x0B4 /* (RO) Support flags for indexed texture formats 32..63 */
+#define TEXTURE_FEATURES_2 0x0B8 /* (RO) Support flags for indexed texture formats 64..95 */
+
+#define TEXTURE_FEATURES_REG(n) GPU_CONTROL_REG(TEXTURE_FEATURES_0 + ((n) << 2))
+
+#define JS0_FEATURES 0x0C0 /* (RO) Features of job slot 0 */
+#define JS1_FEATURES 0x0C4 /* (RO) Features of job slot 1 */
+#define JS2_FEATURES 0x0C8 /* (RO) Features of job slot 2 */
+#define JS3_FEATURES 0x0CC /* (RO) Features of job slot 3 */
+#define JS4_FEATURES 0x0D0 /* (RO) Features of job slot 4 */
+#define JS5_FEATURES 0x0D4 /* (RO) Features of job slot 5 */
+#define JS6_FEATURES 0x0D8 /* (RO) Features of job slot 6 */
+#define JS7_FEATURES 0x0DC /* (RO) Features of job slot 7 */
+#define JS8_FEATURES 0x0E0 /* (RO) Features of job slot 8 */
+#define JS9_FEATURES 0x0E4 /* (RO) Features of job slot 9 */
+#define JS10_FEATURES 0x0E8 /* (RO) Features of job slot 10 */
+#define JS11_FEATURES 0x0EC /* (RO) Features of job slot 11 */
+#define JS12_FEATURES 0x0F0 /* (RO) Features of job slot 12 */
+#define JS13_FEATURES 0x0F4 /* (RO) Features of job slot 13 */
+#define JS14_FEATURES 0x0F8 /* (RO) Features of job slot 14 */
+#define JS15_FEATURES 0x0FC /* (RO) Features of job slot 15 */
+
+#define JS_FEATURES_REG(n) GPU_CONTROL_REG(JS0_FEATURES + ((n) << 2))
+
+#define SHADER_PRESENT_LO 0x100 /* (RO) Shader core present bitmap, low word */
+#define SHADER_PRESENT_HI 0x104 /* (RO) Shader core present bitmap, high word */
+
+#define TILER_PRESENT_LO 0x110 /* (RO) Tiler core present bitmap, low word */
+#define TILER_PRESENT_HI 0x114 /* (RO) Tiler core present bitmap, high word */
+
+#define L2_PRESENT_LO 0x120 /* (RO) Level 2 cache present bitmap, low word */
+#define L2_PRESENT_HI 0x124 /* (RO) Level 2 cache present bitmap, high word */
+
+#define L3_PRESENT_LO 0x130 /* (RO) Level 3 cache present bitmap, low word */
+#define L3_PRESENT_HI 0x134 /* (RO) Level 3 cache present bitmap, high word */
+
+#define SHADER_READY_LO 0x140 /* (RO) Shader core ready bitmap, low word */
+#define SHADER_READY_HI 0x144 /* (RO) Shader core ready bitmap, high word */
+
+#define TILER_READY_LO 0x150 /* (RO) Tiler core ready bitmap, low word */
+#define TILER_READY_HI 0x154 /* (RO) Tiler core ready bitmap, high word */
+
+#define L2_READY_LO 0x160 /* (RO) Level 2 cache ready bitmap, low word */
+#define L2_READY_HI 0x164 /* (RO) Level 2 cache ready bitmap, high word */
+
+#define L3_READY_LO 0x170 /* (RO) Level 3 cache ready bitmap, low word */
+#define L3_READY_HI 0x174 /* (RO) Level 3 cache ready bitmap, high word */
+
+#define SHADER_PWRON_LO 0x180 /* (WO) Shader core power on bitmap, low word */
+#define SHADER_PWRON_HI 0x184 /* (WO) Shader core power on bitmap, high word */
+
+#define TILER_PWRON_LO 0x190 /* (WO) Tiler core power on bitmap, low word */
+#define TILER_PWRON_HI 0x194 /* (WO) Tiler core power on bitmap, high word */
+
+#define L2_PWRON_LO 0x1A0 /* (WO) Level 2 cache power on bitmap, low word */
+#define L2_PWRON_HI 0x1A4 /* (WO) Level 2 cache power on bitmap, high word */
+
+#define L3_PWRON_LO 0x1B0 /* (WO) Level 3 cache power on bitmap, low word */
+#define L3_PWRON_HI 0x1B4 /* (WO) Level 3 cache power on bitmap, high word */
+
+#define SHADER_PWROFF_LO 0x1C0 /* (WO) Shader core power off bitmap, low word */
+#define SHADER_PWROFF_HI 0x1C4 /* (WO) Shader core power off bitmap, high word */
+
+#define TILER_PWROFF_LO 0x1D0 /* (WO) Tiler core power off bitmap, low word */
+#define TILER_PWROFF_HI 0x1D4 /* (WO) Tiler core power off bitmap, high word */
+
+#define L2_PWROFF_LO 0x1E0 /* (WO) Level 2 cache power off bitmap, low word */
+#define L2_PWROFF_HI 0x1E4 /* (WO) Level 2 cache power off bitmap, high word */
+
+#define L3_PWROFF_LO 0x1F0 /* (WO) Level 3 cache power off bitmap, low word */
+#define L3_PWROFF_HI 0x1F4 /* (WO) Level 3 cache power off bitmap, high word */
+
+#define SHADER_PWRTRANS_LO 0x200 /* (RO) Shader core power transition bitmap, low word */
+#define SHADER_PWRTRANS_HI 0x204 /* (RO) Shader core power transition bitmap, high word */
+
+#define TILER_PWRTRANS_LO 0x210 /* (RO) Tiler core power transition bitmap, low word */
+#define TILER_PWRTRANS_HI 0x214 /* (RO) Tiler core power transition bitmap, high word */
+
+#define L2_PWRTRANS_LO 0x220 /* (RO) Level 2 cache power transition bitmap, low word */
+#define L2_PWRTRANS_HI 0x224 /* (RO) Level 2 cache power transition bitmap, high word */
+
+#define L3_PWRTRANS_LO 0x230 /* (RO) Level 3 cache power transition bitmap, low word */
+#define L3_PWRTRANS_HI 0x234 /* (RO) Level 3 cache power transition bitmap, high word */
+
+#define SHADER_PWRACTIVE_LO 0x240 /* (RO) Shader core active bitmap, low word */
+#define SHADER_PWRACTIVE_HI 0x244 /* (RO) Shader core active bitmap, high word */
+
+#define TILER_PWRACTIVE_LO 0x250 /* (RO) Tiler core active bitmap, low word */
+#define TILER_PWRACTIVE_HI 0x254 /* (RO) Tiler core active bitmap, high word */
+
+#define L2_PWRACTIVE_LO 0x260 /* (RO) Level 2 cache active bitmap, low word */
+#define L2_PWRACTIVE_HI 0x264 /* (RO) Level 2 cache active bitmap, high word */
+
+#define L3_PWRACTIVE_LO 0x270 /* (RO) Level 3 cache active bitmap, low word */
+#define L3_PWRACTIVE_HI 0x274 /* (RO) Level 3 cache active bitmap, high word */
+
+
+#define SHADER_CONFIG 0xF04 /* (RW) Shader core configuration settings (Mali-T60x additional register) */
+#define L2_MMU_CONFIG 0xF0C /* (RW) Configuration of the L2 cache and MMU (Mali-T60x additional register) */
+
+
+#define JOB_CONTROL_BASE 0x1000
+
+#define JOB_CONTROL_REG(r) (JOB_CONTROL_BASE + (r))
+
+#define JOB_IRQ_RAWSTAT 0x000 /* Raw interrupt status register */
+#define JOB_IRQ_CLEAR 0x004 /* Interrupt clear register */
+#define JOB_IRQ_MASK 0x008 /* Interrupt mask register */
+#define JOB_IRQ_STATUS 0x00C /* Interrupt status register */
+#define JOB_IRQ_JS_STATE 0x010 /* status==active and _next == busy snapshot from last JOB_IRQ_CLEAR */
+#define JOB_IRQ_THROTTLE 0x014 /* cycles to delay delivering an interrupt externally. The JOB_IRQ_STATUS is NOT affected by this, just the delivery of the interrupt. */
+
+#define JOB_SLOT0 0x800 /* Configuration registers for job slot 0 */
+#define JOB_SLOT1 0x880 /* Configuration registers for job slot 1 */
+#define JOB_SLOT2 0x900 /* Configuration registers for job slot 2 */
+#define JOB_SLOT3 0x980 /* Configuration registers for job slot 3 */
+#define JOB_SLOT4 0xA00 /* Configuration registers for job slot 4 */
+#define JOB_SLOT5 0xA80 /* Configuration registers for job slot 5 */
+#define JOB_SLOT6 0xB00 /* Configuration registers for job slot 6 */
+#define JOB_SLOT7 0xB80 /* Configuration registers for job slot 7 */
+#define JOB_SLOT8 0xC00 /* Configuration registers for job slot 8 */
+#define JOB_SLOT9 0xC80 /* Configuration registers for job slot 9 */
+#define JOB_SLOT10 0xD00 /* Configuration registers for job slot 10 */
+#define JOB_SLOT11 0xD80 /* Configuration registers for job slot 11 */
+#define JOB_SLOT12 0xE00 /* Configuration registers for job slot 12 */
+#define JOB_SLOT13 0xE80 /* Configuration registers for job slot 13 */
+#define JOB_SLOT14 0xF00 /* Configuration registers for job slot 14 */
+#define JOB_SLOT15 0xF80 /* Configuration registers for job slot 15 */
+
+#define JOB_SLOT_REG(n,r) (JOB_CONTROL_REG(JOB_SLOT0 + ((n) << 7)) + (r))
+
+#define JSn_HEAD_LO 0x00 /* (RO) Job queue head pointer for job slot n, low word */
+#define JSn_HEAD_HI 0x04 /* (RO) Job queue head pointer for job slot n, high word */
+#define JSn_TAIL_LO 0x08 /* (RO) Job queue tail pointer for job slot n, low word */
+#define JSn_TAIL_HI 0x0C /* (RO) Job queue tail pointer for job slot n, high word */
+#define JSn_AFFINITY_LO 0x10 /* (RO) Core affinity mask for job slot n, low word */
+#define JSn_AFFINITY_HI 0x14 /* (RO) Core affinity mask for job slot n, high word */
+#define JSn_CONFIG 0x18 /* (RO) Configuration settings for job slot n */
+
+#define JSn_COMMAND 0x20 /* (WO) Command register for job slot n */
+#define JSn_STATUS 0x24 /* (RO) Status register for job slot n */
+
+#define JSn_HEAD_NEXT_LO 0x40 /* (RW) Next job queue head pointer for job slot n, low word */
+#define JSn_HEAD_NEXT_HI 0x44 /* (RW) Next job queue head pointer for job slot n, high word */
+
+#define JSn_AFFINITY_NEXT_LO 0x50 /* (RW) Next core affinity mask for job slot n, low word */
+#define JSn_AFFINITY_NEXT_HI 0x54 /* (RW) Next core affinity mask for job slot n, high word */
+#define JSn_CONFIG_NEXT 0x58 /* (RW) Next configuration settings for job slot n */
+
+#define JSn_COMMAND_NEXT 0x60 /* (RW) Next command register for job slot n */
+
+
+#define MEMORY_MANAGEMENT_BASE 0x2000
+#define MMU_REG(r) (MEMORY_MANAGEMENT_BASE + (r))
+
+#define MMU_IRQ_RAWSTAT 0x000 /* (RW) Raw interrupt status register */
+#define MMU_IRQ_CLEAR 0x004 /* (WO) Interrupt clear register */
+#define MMU_IRQ_MASK 0x008 /* (RW) Interrupt mask register */
+#define MMU_IRQ_STATUS 0x00C /* (RO) Interrupt status register */
+
+#define MMU_AS0 0x400 /* Configuration registers for address space 0 */
+#define MMU_AS1 0x440 /* Configuration registers for address space 1 */
+#define MMU_AS2 0x480 /* Configuration registers for address space 2 */
+#define MMU_AS3 0x4C0 /* Configuration registers for address space 3 */
+#define MMU_AS4 0x500 /* Configuration registers for address space 4 */
+#define MMU_AS5 0x540 /* Configuration registers for address space 5 */
+#define MMU_AS6 0x580 /* Configuration registers for address space 6 */
+#define MMU_AS7 0x5C0 /* Configuration registers for address space 7 */
+#define MMU_AS8 0x600 /* Configuration registers for address space 8 */
+#define MMU_AS9 0x640 /* Configuration registers for address space 9 */
+#define MMU_AS10 0x680 /* Configuration registers for address space 10 */
+#define MMU_AS11 0x6C0 /* Configuration registers for address space 11 */
+#define MMU_AS12 0x700 /* Configuration registers for address space 12 */
+#define MMU_AS13 0x740 /* Configuration registers for address space 13 */
+#define MMU_AS14 0x780 /* Configuration registers for address space 14 */
+#define MMU_AS15 0x7C0 /* Configuration registers for address space 15 */
+
+#define MMU_AS_REG(n,r) (MMU_REG(MMU_AS0 + ((n) << 6)) + (r))
+
+#define ASn_TRANSTAB_LO 0x00 /* (RW) Translation Table Base Address for address space n, low word */
+#define ASn_TRANSTAB_HI 0x04 /* (RW) Translation Table Base Address for address space n, high word */
+#define ASn_MEMATTR_LO 0x08 /* (RW) Memory attributes for address space n, low word. */
+#define ASn_MEMATTR_HI 0x0C /* (RW) Memory attributes for address space n, high word. */
+#define ASn_LOCKADDR_LO 0x10 /* (RW) Lock region address for address space n, low word */
+#define ASn_LOCKADDR_HI 0x14 /* (RW) Lock region address for address space n, high word */
+#define ASn_COMMAND 0x18 /* (WO) MMU command register for address space n */
+#define ASn_FAULTSTATUS 0x1C /* (RO) MMU fault status register for address space n */
+#define ASn_FAULTADDRESS_LO 0x20 /* (RO) Fault Address for address space n, low word */
+#define ASn_FAULTADDRESS_HI 0x24 /* (RO) Fault Address for address space n, high word */
+#define ASn_STATUS 0x28 /* (RO) Status flags for address space n */
+
+/* End Register Offsets */
+
+/*
+ * MMU_IRQ_RAWSTAT register values. Values are valid also for
+ MMU_IRQ_CLEAR, MMU_IRQ_MASK, MMU_IRQ_STATUS registers.
+ */
+
+#define MMU_REGS_PAGE_FAULT_FLAGS 16
+
+/* Macros return bit number to retrvie page fault or bus eror flag from MMU registers */
+#define MMU_REGS_PAGE_FAULT_FLAG(n) (n)
+#define MMU_REGS_BUS_ERROR_FLAG(n) (n + MMU_REGS_PAGE_FAULT_FLAGS)
+
+/*
+ * Begin MMU TRANSTAB register values
+ */
+#define ASn_TRANSTAB_ADDR_SPACE_MASK 0xfffff000
+#define ASn_TRANSTAB_ADRMODE_UNMAPPED (0u << 0)
+#define ASn_TRANSTAB_ADRMODE_IDENTITY (1u << 1)
+#define ASn_TRANSTAB_ADRMODE_TABLE (3u << 0)
+#define ASn_TRANSTAB_READ_INNER (1u << 2)
+#define ASn_TRANSTAB_SHARE_OUTER (1u << 4)
+
+#define MMU_TRANSTAB_ADRMODE_MASK 0x00000003
+
+/*
+ * Begin MMU STATUS register values
+ */
+#define ASn_STATUS_FLUSH_ACTIVE 0x01
+
+#define ASn_FAULTSTATUS_ACCESS_TYPE_MASK (0x3<<8)
+#define ASn_FAULTSTATUS_ACCESS_TYPE_EX (0x1<<8)
+#define ASn_FAULTSTATUS_ACCESS_TYPE_READ (0x2<<8)
+#define ASn_FAULTSTATUS_ACCESS_TYPE_WRITE (0x3<<8)
+
+/*
+ * Begin Command Values
+ */
+
+/* JSn_COMMAND register commands */
+#define JSn_COMMAND_NOP 0x00 /* NOP Operation. Writing this value is ignored */
+#define JSn_COMMAND_START 0x01 /* Start processing a job chain. Writing this value is ignored */
+#define JSn_COMMAND_SOFT_STOP 0x02 /* Gently stop processing a job chain */
+#define JSn_COMMAND_HARD_STOP 0x03 /* Rudely stop processing a job chain */
+
+/* ASn_COMMAND register commands */
+#define ASn_COMMAND_NOP 0x00 /* NOP Operation */
+#define ASn_COMMAND_UPDATE 0x01 /* Broadcasts the values in ASn_TRANSTAB and ASn_MEMATTR to all MMUs */
+#define ASn_COMMAND_LOCK 0x02 /* Issue a lock region command to all MMUs */
+#define ASn_COMMAND_UNLOCK 0x03 /* Issue a flush region command to all MMUs */
+#define ASn_COMMAND_FLUSH 0x04 /* Flush all L2 caches then issue a flush region command to all MMUs */
+
+/* Possible values of JSn_CONFIG and JSn_CONFIG_NEXT registers */
+#define JSn_CONFIG_START_FLUSH_NO_ACTION (0u << 0)
+#define JSn_CONFIG_START_FLUSH_CLEAN (1u << 8)
+#define JSn_CONFIG_START_FLUSH_CLEAN_INVALIDATE (3u << 8)
+#define JSn_CONFIG_START_MMU (1u << 10)
+#define JSn_CONFIG_END_FLUSH_NO_ACTION JSn_CONFIG_START_FLUSH_NO_ACTION
+#define JSn_CONFIG_END_FLUSH_CLEAN (1u << 12)
+#define JSn_CONFIG_END_FLUSH_CLEAN_INVALIDATE (3u << 12)
+#define JSn_CONFIG_THREAD_PRI(n) ((n) << 16)
+
+/* JSn_STATUS register values */
+
+/* NOTE: Please keep this values in sync with enum base_jd_event_code in mali_base_kernel.h.
+ * The values are separated to avoid dependency of userspace and kernel code.
+ */
+
+/* Group of values representing the job status insead a particular fault */
+#define JSn_STATUS_NO_EXCEPTION_BASE 0x00
+#define JSn_STATUS_INTERRUPTED (JSn_STATUS_NO_EXCEPTION_BASE + 0x02) /* 0x02 means INTERRUPTED */
+#define JSn_STATUS_STOPPED (JSn_STATUS_NO_EXCEPTION_BASE + 0x03) /* 0x03 means STOPPED */
+#define JSn_STATUS_TERMINATED (JSn_STATUS_NO_EXCEPTION_BASE + 0x04) /* 0x04 means TERMINATED */
+
+/* General fault values */
+#define JSn_STATUS_FAULT_BASE 0x40
+#define JSn_STATUS_CONFIG_FAULT (JSn_STATUS_FAULT_BASE) /* 0x40 means CONFIG FAULT */
+#define JSn_STATUS_POWER_FAULT (JSn_STATUS_FAULT_BASE + 0x01) /* 0x41 means POWER FAULT */
+#define JSn_STATUS_READ_FAULT (JSn_STATUS_FAULT_BASE + 0x02) /* 0x42 means READ FAULT */
+#define JSn_STATUS_WRITE_FAULT (JSn_STATUS_FAULT_BASE + 0x03) /* 0x43 means WRITE FAULT */
+#define JSn_STATUS_AFFINITY_FAULT (JSn_STATUS_FAULT_BASE + 0x04) /* 0x44 means AFFINITY FAULT */
+#define JSn_STATUS_BUS_FAULT (JSn_STATUS_FAULT_BASE + 0x08) /* 0x48 means BUS FAULT */
+
+/* Instruction or data faults */
+#define JSn_STATUS_INSTRUCTION_FAULT_BASE 0x50
+#define JSn_STATUS_INSTR_INVALID_PC (JSn_STATUS_INSTRUCTION_FAULT_BASE) /* 0x50 means INSTR INVALID PC */
+#define JSn_STATUS_INSTR_INVALID_ENC (JSn_STATUS_INSTRUCTION_FAULT_BASE + 0x01) /* 0x51 means INSTR INVALID ENC */
+#define JSn_STATUS_INSTR_TYPE_MISMATCH (JSn_STATUS_INSTRUCTION_FAULT_BASE + 0x02) /* 0x52 means INSTR TYPE MISMATCH */
+#define JSn_STATUS_INSTR_OPERAND_FAULT (JSn_STATUS_INSTRUCTION_FAULT_BASE + 0x03) /* 0x53 means INSTR OPERAND FAULT */
+#define JSn_STATUS_INSTR_TLS_FAULT (JSn_STATUS_INSTRUCTION_FAULT_BASE + 0x04) /* 0x54 means INSTR TLS FAULT */
+#define JSn_STATUS_INSTR_BARRIER_FAULT (JSn_STATUS_INSTRUCTION_FAULT_BASE + 0x05) /* 0x55 means INSTR BARRIER FAULT */
+#define JSn_STATUS_INSTR_ALIGN_FAULT (JSn_STATUS_INSTRUCTION_FAULT_BASE + 0x06) /* 0x56 means INSTR ALIGN FAULT */
+/* NOTE: No fault with 0x57 code defined in spec. */
+#define JSn_STATUS_DATA_INVALID_FAULT (JSn_STATUS_INSTRUCTION_FAULT_BASE + 0x08) /* 0x58 means DATA INVALID FAULT */
+#define JSn_STATUS_TILE_RANGE_FAULT (JSn_STATUS_INSTRUCTION_FAULT_BASE + 0x09) /* 0x59 means TILE RANGE FAULT */
+#define JSn_STATUS_ADDRESS_RANGE_FAULT (JSn_STATUS_INSTRUCTION_FAULT_BASE + 0x0A) /* 0x5A means ADDRESS RANGE FAULT */
+
+/* Other faults */
+#define JSn_STATUS_MEMORY_FAULT_BASE 0x60
+#define JSn_STATUS_OUT_OF_MEMORY (JSn_STATUS_MEMORY_FAULT_BASE) /* 0x60 means OUT OF MEMORY */
+#define JSn_STATUS_UNKNOWN 0x7F /* 0x7F means UNKNOWN */
+
+
+/* GPU_COMMAND values */
+#define GPU_COMMAND_NOP 0x00 /* No operation, nothing happens */
+#define GPU_COMMAND_SOFT_RESET 0x01 /* Stop all external bus interfaces, and then reset the entire GPU. */
+#define GPU_COMMAND_HARD_RESET 0x02 /* Immediately reset the entire GPU. */
+#define GPU_COMMAND_PRFCNT_CLEAR 0x03 /* Clear all performance counters, setting them all to zero. */
+#define GPU_COMMAND_PRFCNT_SAMPLE 0x04 /* Sample all performance counters, writing them out to memory */
+#define GPU_COMMAND_CYCLE_COUNT_START 0x05 /* Starts the cycle counter, and system timestamp propagation */
+#define GPU_COMMAND_CYCLE_COUNT_STOP 0x06 /* Stops the cycle counter, and system timestamp propagation */
+#define GPU_COMMAND_CLEAN_CACHES 0x07 /* Clean all caches */
+#define GPU_COMMAND_CLEAN_INV_CACHES 0x08 /* Clean and invalidate all caches */
+
+/* End Command Values */
+
+/* GPU_STATUS values */
+#define GPU_STATUS_PRFCNT_ACTIVE (1 << 2) /* Set if the performance counters are active. */
+
+/* PRFCNT_CONFIG register values */
+#define PRFCNT_CONFIG_AS_SHIFT 4 /* address space bitmap starts from bit 4 of the register */
+#define PRFCNT_CONFIG_MODE_OFF 0 /* The performance counters are disabled. */
+#define PRFCNT_CONFIG_MODE_MANUAL 1 /* The performance counters are enabled, but are only written out when a PRFCNT_SAMPLE command is issued using the GPU_COMMAND register. */
+#define PRFCNT_CONFIG_MODE_TILE 2 /* The performance counters are enabled, and are written out each time a tile finishes rendering. */
+
+/* AS<n>_MEMATTR values */
+#define ASn_MEMATTR_IMPL_DEF_CACHE_POLICY 0x48484848 /* Use GPU implementation-defined caching policy. */
+#define ASn_MEMATTR_FORCE_TO_CACHE_ALL 0x4F4F4F4F /* The attribute set to force all resources to be cached. */
+
+/* GPU_ID register */
+#define GPU_ID_VERSION_STATUS_SHIFT 0
+#define GPU_ID_VERSION_MINOR_SHIFT 4
+#define GPU_ID_VERSION_MAJOR_SHIFT 12
+#define GPU_ID_VERSION_PRODUCT_ID_SHIFT 16
+#define GPU_ID_VERSION_STATUS (0xF << GPU_ID_VERSION_STATUS_SHIFT)
+#define GPU_ID_VERSION_MINOR (0xFF << GPU_ID_VERSION_MINOR_SHIFT)
+#define GPU_ID_VERSION_MAJOR (0xF << GPU_ID_VERSION_MAJOR_SHIFT)
+#define GPU_ID_VERSION_PRODUCT_ID (0xFFFF << GPU_ID_VERSION_PRODUCT_ID_SHIFT)
+
+/* Values for GPU_ID_VERSION_PRODUCT_ID bitfield */
+#define GPU_ID_PI_T60X 0x6956
+#define GPU_ID_PI_T65X 0x3456
+#define GPU_ID_PI_T62X 0x0620
+#define GPU_ID_PI_T67X 0x0670
+
+/* Values for GPU_ID_VERSION_STATUS field for PRODUCT_ID GPU_ID_PI_T60X and GPU_ID_PI_T65X */
+#define GPU_ID_S_15DEV0 0x1
+#define GPU_ID_S_EAC 0x2
+
+/* Helper macro to create a GPU_ID assuming valid values for id, major, minor, status */
+#define GPU_ID_MAKE(id, major, minor, status) \
+ (((id) << GPU_ID_VERSION_PRODUCT_ID_SHIFT) | \
+ ((major) << GPU_ID_VERSION_MAJOR_SHIFT) | \
+ ((minor) << GPU_ID_VERSION_MINOR_SHIFT) | \
+ ((status) << GPU_ID_VERSION_STATUS_SHIFT))
+
+/* End GPU_ID register */
+
+/* JS<n>_FEATURES register */
+
+#define JSn_FEATURE_NULL_JOB (1u << 1)
+#define JSn_FEATURE_SET_VALUE_JOB (1u << 2)
+#define JSn_FEATURE_CACHE_FLUSH_JOB (1u << 3)
+#define JSn_FEATURE_COMPUTE_JOB (1u << 4)
+#define JSn_FEATURE_VERTEX_JOB (1u << 5)
+#define JSn_FEATURE_GEOMETRY_JOB (1u << 6)
+#define JSn_FEATURE_TILER_JOB (1u << 7)
+#define JSn_FEATURE_FUSED_JOB (1u << 8)
+#define JSn_FEATURE_FRAGMENT_JOB (1u << 9)
+
+/* End JS<n>_FEATURES register */
+
+#endif /* _MIDGARD_REGMAP_H_ */
--- /dev/null
+ccflags-$(CONFIG_VITHAR) += -DMALI_DEBUG=1 -DMALI_HW_TYPE=2 \
+-DMALI_USE_UMP=0 -DMALI_HW_VERSION=r0p0 -DMALI_BASE_TRACK_MEMLEAK=0 \
+-DMALI_ANDROID=1 -DMALI_ERROR_INJECT_ON=0 -DMALI_NO_MALI=0 -DMALI_BACKEND_KERNEL=1 \
+-DMALI_FAKE_PLATFORM_DEVICE=1 -DMALI_MOCK_TEST=0 -DMALI_KERNEL_TEST_API=0 \
+-DMALI_INFINITE_CACHE=0 -DMALI_LICENSE_IS_GPL=1 -DMALI_PLATFORM_CONFIG=exynos5 \
+-DMALI_UNIT_TEST=0 -DMALI_GATOR_SUPPORT=0 -DMALI_LICENSE_IS_GPL=1 \
+-DUMP_SVN_REV_STRING="\"dummy\"" -DMALI_RELEASE_NAME="\"dummy\""
+
+ROOTDIR = $(src)/../../..
+
+ccflags-$(CONFIG_VITHAR) += -I$(ROOTDIR)/kbase
+ccflags-$(CONFIG_VITHAR) += -I$(ROOTDIR) -I$(src)/..
+
+ccflags-y += -I$(ROOTDIR) -I$(ROOTDIR)/include -I$(ROOTDIR)/osk/src/linux/include -I$(ROOTDIR)/uk/platform_dummy
+ccflags-y += -I$(ROOTDIR)/kbase/midg_gpus/r0p0
+
+
+obj-y += mali_kbase_mem_linux.o
+obj-y += mali_kbase_core_linux.o
+obj-y += mali_kbase_config_linux.o
+
+obj-y += config/
--- /dev/null
+ccflags-$(CONFIG_VITHAR) += -DMALI_DEBUG=1 -DMALI_HW_TYPE=2 \
+-DMALI_USE_UMP=0 -DMALI_HW_VERSION=r0p0 -DMALI_BASE_TRACK_MEMLEAK=0 \
+-DMALI_ANDROID=1 -DMALI_ERROR_INJECT_ON=0 -DMALI_NO_MALI=0 -DMALI_BACKEND_KERNEL=1 \
+-DMALI_FAKE_PLATFORM_DEVICE=1 -DMALI_MOCK_TEST=0 -DMALI_KERNEL_TEST_API=0 \
+-DMALI_INFINITE_CACHE=0 -DMALI_LICENSE_IS_GPL=1 -DMALI_PLATFORM_CONFIG=exynos5 \
+-DMALI_UNIT_TEST=0 -DMALI_GATOR_SUPPORT=0 -DMALI_LICENSE_IS_GPL=1 \
+-DUMP_SVN_REV_STRING="\"dummy\"" -DMALI_RELEASE_NAME="\"dummy\"" \
+-DMALI_CUSTOMER_RELEASE=1 -DMALI_UNCACHED=1
+
+ROOTDIR = $(src)/../../../..
+
+ccflags-$(CONFIG_VITHAR) += -I$(ROOTDIR)/kbase
+ccflags-$(CONFIG_VITHAR) += -I$(ROOTDIR) -I$(src)/..
+
+ccflags-y += -I$(ROOTDIR) -I$(ROOTDIR)/include -I$(ROOTDIR)/osk/src/linux/include -I$(ROOTDIR)/uk/platform_dummy -I$(ROOTDIR)/kbase/midg_gpus/r0p0
+
+obj-y += mali_kbase_config_exynos5.o
--- /dev/null
+/* LICENSE NOT CONFIRMED. Do not distribute without permission. Assume confidential and proprietary until classification is authorised. */
+/*
+ *
+ * (C) COPYRIGHT 2012 ARM Limited. All rights reserved.
+ *
+ * This program is free software and is provided to you under the terms of the GNU General Public License version 2
+ * as published by the Free Software Foundation, and any use by you of this program is subject to the terms of such GNU licence.
+ *
+ * A copy of the licence is included with the program, and can also be obtained from Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <linux/platform_device.h>
+#include <linux/pci.h>
+#include <linux/miscdevice.h>
+#include <linux/list.h>
+#include <linux/semaphore.h>
+#include <linux/fs.h>
+#include <linux/uaccess.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/ioport.h>
+#include <linux/spinlock.h>
+
+#include <mach/map.h>
+#include <linux/fb.h>
+#include <linux/clk.h>
+#include <mach/regs-clock.h>
+#include <mach/pmu.h>
+#include <mach/regs-pmu.h>
+#include <asm/delay.h>
+#include <mach/map.h>
+#include <generated/autoconf.h>
+
+#include <linux/timer.h>
+#include <linux/pm_runtime.h>
+#include <linux/workqueue.h>
+#include <linux/regulator/consumer.h>
+#include <linux/regulator/driver.h>
+
+#include <osk/mali_osk.h>
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_kbase_pm.h>
+#include <kbase/src/common/mali_kbase_uku.h>
+#include <kbase/src/common/mali_kbase_mem.h>
+#include <kbase/src/common/mali_midg_regmap.h>
+#include <kbase/src/linux/mali_kbase_mem_linux.h>
+#include <uk/mali_ukk.h>
+#include <kbase/src/common/mali_kbase_defs.h>
+#include <kbase/src/linux/mali_kbase_config_linux.h>
+#if MALI_USE_UMP == 1
+#include <ump/ump_common.h>
+#endif /*MALI_USE_UMP == 1*/
+
+#if MALI_UNCACHED == 0
+#error MALI_UNCACHED should equal 1 for Exynos5 support, your scons commandline should contain 'no_syncsets=1'
+#endif
+
+#define HZ_IN_MHZ (1000000)
+#define MALI_RTPM_DEBUG 0
+#define VITHAR_DEFAULT_CLOCK 533000000
+#define RUNTIME_PM_DELAY_TIME 10
+
+struct regulator *kbase_platform_get_regulator(void);
+int kbase_platform_regulator_init(void);
+int kbase_platform_regulator_disable(void);
+int kbase_platform_regulator_enable(struct device *dev);
+int kbase_platform_get_default_voltage(struct device *dev, int *vol);
+int kbase_platform_get_voltage(struct device *dev, int *vol);
+int kbase_platform_set_voltage(struct device *dev, int vol);
+void kbase_platform_dvfs_set_clock(kbase_device *kbdev, int freq);
+
+#ifdef CONFIG_VITHAR_DVFS
+int kbase_platform_dvfs_init(kbase_device *kbdev);
+void kbase_platform_dvfs_term(void);
+int kbase_platform_dvfs_event(kbase_device *kbdev, u32 utilisation);
+int kbase_platform_dvfs_get_control_status(void);
+int kbase_pm_get_dvfs_utilisation(kbase_device *kbdev);
+#ifdef CONFIG_VITHAR_FREQ_LOCK
+int mali_get_dvfs_upper_locked_freq(void);
+int mali_get_dvfs_under_locked_freq(void);
+int mali_dvfs_freq_lock(int level);
+void mali_dvfs_freq_unlock(void);
+int mali_dvfs_freq_under_lock(int level);
+void mali_dvfs_freq_under_unlock(void);
+#endif /* CONFIG_VITHAR_FREQ_LOCK */
+#endif /* CONFIG_VITHAR_DVFS */
+
+#ifdef CONFIG_PM_RUNTIME
+static void kbase_platform_runtime_term(struct kbase_device *kbdev);
+static mali_error kbase_platform_runtime_init(struct kbase_device *kbdev);
+#endif /* CONFIG_PM_RUNTIME */
+
+int kbase_platform_cmu_pmu_control(struct kbase_device *kbdev, int control);
+void kbase_platform_remove_sysfs_file(struct device *dev);
+mali_error kbase_platform_init(struct kbase_device *kbdev);
+static int kbase_platform_is_power_on(void);
+void kbase_platform_term(struct kbase_device *kbdev);
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static int kbase_platform_create_sysfs_file(struct device *dev);
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+struct exynos_context
+{
+ /** Indicator if system clock to mail-t604 is active */
+ int cmu_pmu_status;
+ /** cmd & pmu lock */
+ spinlock_t cmu_pmu_lock;
+ struct clk *sclk_g3d;
+};
+
+static kbase_io_resources io_resources =
+{
+ .job_irq_number = EXYNOS5_JOB_IRQ_NUMBER,
+ .mmu_irq_number = EXYNOS5_MMU_IRQ_NUMBER,
+ .gpu_irq_number = EXYNOS5_GPU_IRQ_NUMBER,
+ .io_memory_region =
+ {
+ .start = EXYNOS5_PA_G3D,
+ .end = EXYNOS5_PA_G3D + (4096 * 5) - 1
+ }
+};
+
+/**
+ * Read the CPU clock speed
+ */
+int get_cpu_clock_speed(u32* cpu_clock)
+{
+ struct clk * cpu_clk;
+ u32 freq=0;
+ cpu_clk = clk_get(NULL, "armclk");
+ if (IS_ERR(cpu_clk))
+ return 1;
+ freq = clk_get_rate(cpu_clk);
+ *cpu_clock = (freq/HZ_IN_MHZ);
+ return 0;
+}
+
+/**
+ * Power Management callback - power ON
+ */
+static int pm_callback_power_on(kbase_device *kbdev)
+{
+#ifdef CONFIG_PM_RUNTIME
+ pm_runtime_resume(kbdev->osdev.dev);
+#endif /* CONFIG_PM_RUNTIME */
+ return 0;
+}
+
+/**
+ * Power Management callback - power OFF
+ */
+static void pm_callback_power_off(kbase_device *kbdev)
+{
+#ifdef CONFIG_PM_RUNTIME
+ pm_schedule_suspend(kbdev->osdev.dev, RUNTIME_PM_DELAY_TIME);
+#endif /* CONFIG_PM_RUNTIME */
+}
+
+/**
+ * Power Management callback - runtime power ON
+ */
+#ifdef CONFIG_PM_RUNTIME
+static int pm_callback_runtime_power_on(kbase_device *kbdev)
+{
+#if MALI_RTPM_DEBUG
+ printk("kbase_device_runtime_resume\n");
+#endif /* MALI_RTPM_DEBUG */
+ return kbase_platform_cmu_pmu_control(kbdev, 1);
+}
+#endif /* CONFIG_PM_RUNTIME */
+
+/**
+ * Power Management callback - runtime power OFF
+ */
+#ifdef CONFIG_PM_RUNTIME
+static void pm_callback_runtime_power_off(kbase_device *kbdev)
+{
+#if MALI_RTPM_DEBUG
+ printk("kbase_device_runtime_suspend\n");
+#endif /* MALI_RTPM_DEBUG */
+ kbase_platform_cmu_pmu_control(kbdev, 0);
+}
+#endif /* CONFIG_PM_RUNTIME */
+
+static kbase_pm_callback_conf pm_callbacks =
+{
+ .power_on_callback = pm_callback_power_on,
+ .power_off_callback = pm_callback_power_off,
+};
+
+/**
+ * Exynos5 hardware specific initialization
+ */
+mali_bool kbase_platform_exynos5_init(kbase_device *kbdev)
+{
+ if(MALI_ERROR_NONE == kbase_platform_init(kbdev))
+ {
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+ if(kbase_platform_create_sysfs_file(kbdev->osdev.dev))
+ {
+ return MALI_TRUE;
+ }
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+ return MALI_TRUE;
+ }
+
+ return MALI_FALSE;
+}
+
+/**
+ * Exynos5 hardware specific termination
+ */
+void kbase_platform_exynos5_term(kbase_device *kbdev)
+{
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+ kbase_platform_remove_sysfs_file(kbdev->osdev.dev);
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+ kbase_platform_term(kbdev);
+}
+
+kbase_platform_funcs_conf platform_funcs =
+{
+ .platform_init_func = &kbase_platform_exynos5_init,
+ .platform_term_func = &kbase_platform_exynos5_term,
+};
+
+static kbase_attribute config_attributes[] = {
+#if MALI_USE_UMP == 1
+ {
+ KBASE_CONFIG_ATTR_UMP_DEVICE,
+ UMP_DEVICE_Z_SHIFT
+ },
+#endif /* MALI_USE_UMP == 1 */
+ {
+ KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_MAX,
+ 2048 * 1024 * 1024UL /* 2048MB */
+ },
+
+ {
+ KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_PERF_GPU,
+ KBASE_MEM_PERF_FAST
+ },
+ {
+ KBASE_CONFIG_ATTR_POWER_MANAGEMENT_CALLBACKS,
+ (uintptr_t)&pm_callbacks
+ },
+ {
+ KBASE_CONFIG_ATTR_PLATFORM_FUNCS,
+ (uintptr_t)&platform_funcs
+ },
+
+ {
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MAX,
+ 533000
+ },
+ {
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MIN,
+ 100000
+ },
+ {
+ KBASE_CONFIG_ATTR_JS_RESET_TIMEOUT_MS,
+ 500 /* 500ms before cancelling stuck jobs */
+ },
+ {
+ KBASE_CONFIG_ATTR_CPU_SPEED_FUNC,
+ (uintptr_t)&get_cpu_clock_speed
+ },
+ {
+ KBASE_CONFIG_ATTR_END,
+ 0
+ }
+};
+
+kbase_platform_config platform_config =
+{
+ .attributes = config_attributes,
+ .io_resources = &io_resources,
+ .midgard_type = KBASE_MALI_T604
+};
+
+static struct clk *clk_g3d = NULL;
+
+/**
+ * Initialize GPU clocks
+ */
+static int kbase_platform_power_clock_init(kbase_device *kbdev)
+{
+ struct device *dev = kbdev->osdev.dev;
+ int timeout;
+ struct exynos_context *platform;
+
+ platform = (struct exynos_context *) kbdev->platform_context;
+ if(NULL == platform)
+ {
+ panic("oops");
+ }
+
+ /* Turn on G3D power */
+ __raw_writel(0x7, EXYNOS5_G3D_CONFIGURATION);
+
+ /* Wait for G3D power stability for 1ms */
+ timeout = 10;
+ while((__raw_readl(EXYNOS5_G3D_STATUS) & 0x7) != 0x7) {
+ if(timeout == 0) {
+ /* need to call panic */
+ panic("failed to turn on g3d power\n");
+ goto out;
+ }
+ timeout--;
+ udelay(100);
+ }
+
+ /* Turn on G3D clock */
+ clk_g3d = clk_get(dev, "g3d");
+ if(IS_ERR(clk_g3d)) {
+ OSK_PRINT_ERROR(OSK_BASE_PM, "failed to clk_get [clk_g3d]\n");
+ /* chrome linux does not have this clock */
+ }
+ else
+ {
+ /* android_v4 support */
+ clk_enable(clk_g3d);
+ printk("v4 support\n");
+ }
+
+#ifdef CONFIG_VITHAR_HWVER_R0P0
+ platform->sclk_g3d = clk_get(dev, "aclk_400");
+ if(IS_ERR(platform->sclk_g3d)) {
+ OSK_PRINT_ERROR(OSK_BASE_PM, "failed to clk_get [sclk_g3d]\n");
+ goto out;
+ }
+#else /* CONFIG_VITHAR_HWVER_R0P0 */
+ {
+ struct clk *mpll = NULL;
+ mpll = clk_get(dev, "mout_mpll_user");
+ if(IS_ERR(mpll)) {
+ OSK_PRINT_ERROR(OSK_BASE_PM, "failed to clk_get [mout_mpll_user]\n");
+ goto out;
+ }
+
+ platform->sclk_g3d = clk_get(dev, "sclk_g3d");
+ if(IS_ERR(platform->sclk_g3d)) {
+ OSK_PRINT_ERROR(OSK_BASE_PM, "failed to clk_get [sclk_g3d]\n");
+ goto out;
+ }
+
+ clk_set_parent(platform->sclk_g3d, mpll);
+ if(IS_ERR(platform->sclk_g3d)) {
+ OSK_PRINT_ERROR(OSK_BASE_PM, "failed to clk_set_parent\n");
+ goto out;
+ }
+
+ clk_set_rate(platform->sclk_g3d, VITHAR_DEFAULT_CLOCK);
+ if(IS_ERR(platform->sclk_g3d)) {
+ OSK_PRINT_ERROR(OSK_BASE_PM, "failed to clk_set_rate [sclk_g3d] = %d\n", VITHAR_DEFAULT_CLOCK);
+ goto out;
+ }
+ }
+#endif /* CONFIG_VITHAR_HWVER_R0P0 */
+ (void) clk_enable(platform->sclk_g3d);
+ return 0;
+out:
+ return -EPERM;
+}
+
+/**
+ * Enable GPU clocks
+ */
+static int kbase_platform_clock_on(struct kbase_device *kbdev)
+{
+ struct exynos_context *platform;
+ if (!kbdev)
+ return -ENODEV;
+
+ platform = (struct exynos_context *) kbdev->platform_context;
+ if (!platform)
+ return -ENODEV;
+
+ if(clk_g3d)
+ {
+ /* android_v4 support */
+ (void) clk_enable(clk_g3d);
+ }
+ else
+ {
+ /* chrome support */
+ (void) clk_enable(platform->sclk_g3d);
+ }
+
+ return 0;
+}
+
+/**
+ * Disable GPU clocks
+ */
+static int kbase_platform_clock_off(struct kbase_device *kbdev)
+{
+ struct exynos_context *platform;
+ if (!kbdev)
+ return -ENODEV;
+
+ platform = (struct exynos_context *) kbdev->platform_context;
+ if (!platform)
+ return -ENODEV;
+
+ if(clk_g3d)
+ {
+ /* android_v4 support */
+ (void)clk_disable(clk_g3d);
+ }
+ else
+ {
+ /* chrome support */
+ (void)clk_disable(platform->sclk_g3d);
+ }
+ return 0;
+}
+
+/**
+ * Report GPU power status
+ */
+static inline int kbase_platform_is_power_on(void)
+{
+ return ((__raw_readl(EXYNOS5_G3D_STATUS) & 0x7) == 0x7) ? 1 : 0;
+}
+
+/**
+ * Enable GPU power
+ */
+static int kbase_platform_power_on(void)
+{
+ int timeout;
+
+ /* Turn on G3D */
+ __raw_writel(0x7, EXYNOS5_G3D_CONFIGURATION);
+
+ /* Wait for G3D power stability */
+ timeout = 1000;
+
+ while((__raw_readl(EXYNOS5_G3D_STATUS) & 0x7) != 0x7) {
+ if(timeout == 0) {
+ /* need to call panic */
+ panic("failed to turn on g3d via g3d_configuration\n");
+ return -ETIMEDOUT;
+ }
+ timeout--;
+ udelay(10);
+ }
+
+ return 0;
+}
+
+/**
+ * Disable GPU power
+ */
+static int kbase_platform_power_off(void)
+{
+ int timeout;
+
+ /* Turn off G3D */
+ __raw_writel(0x0, EXYNOS5_G3D_CONFIGURATION);
+
+ /* Wait for G3D power stability */
+ timeout = 1000;
+
+ while(__raw_readl(EXYNOS5_G3D_STATUS) & 0x7) {
+ if(timeout == 0) {
+ /* need to call panic */
+ panic( "failed to turn off g3d via g3d_configuration\n");
+ return -ETIMEDOUT;
+ }
+ timeout--;
+ udelay(10);
+ }
+
+ return 0;
+}
+
+/**
+ * Power Management unit control. Enable/disable power and clocks to GPU
+ */
+int kbase_platform_cmu_pmu_control(struct kbase_device *kbdev, int control)
+{
+ unsigned long flags;
+ struct exynos_context *platform;
+ if (!kbdev)
+ {
+ return -ENODEV;
+ }
+
+ platform = (struct exynos_context *) kbdev->platform_context;
+ if (!platform)
+ {
+ return -ENODEV;
+ }
+
+ spin_lock_irqsave(&platform->cmu_pmu_lock, flags);
+
+ /* off */
+ if(control == 0)
+ {
+ if(platform->cmu_pmu_status == 0)
+ {
+ spin_unlock_irqrestore(&platform->cmu_pmu_lock, flags);
+ return 0;
+ }
+
+ if(kbase_platform_power_off())
+ panic("failed to turn off g3d power\n");
+ if(kbase_platform_clock_off(kbdev))
+
+ panic("failed to turn off sclk_g3d\n");
+
+ platform->cmu_pmu_status = 0;
+#if MALI_RTPM_DEBUG
+ printk( KERN_ERR "3D cmu_pmu_control - off\n" );
+#endif /* MALI_RTPM_DEBUG */
+ }
+ else
+ {
+ /* on */
+ if(platform->cmu_pmu_status == 1)
+ {
+ spin_unlock_irqrestore(&platform->cmu_pmu_lock, flags);
+ return 0;
+ }
+
+ if(kbase_platform_clock_on(kbdev))
+ panic("failed to turn on sclk_g3d\n");
+ if(kbase_platform_power_on())
+ panic("failed to turn on g3d power\n");
+
+ platform->cmu_pmu_status = 1;
+#if MALI_RTPM_DEBUG
+ printk( KERN_ERR "3D cmu_pmu_control - on\n");
+#endif /* MALI_RTPM_DEBUG */
+ }
+
+ spin_unlock_irqrestore(&platform->cmu_pmu_lock, flags);
+
+ return 0;
+}
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static ssize_t show_clock(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct kbase_device *kbdev;
+ struct exynos_context *platform;
+ ssize_t ret = 0;
+ unsigned int clkrate;
+
+ kbdev = dev_get_drvdata(dev);
+
+ if (!kbdev)
+ return -ENODEV;
+
+ platform = (struct exynos_context *) kbdev->platform_context;
+ if(!platform)
+ return -ENODEV;
+
+ if(!platform->sclk_g3d)
+ return -ENODEV;
+
+ clkrate = clk_get_rate(platform->sclk_g3d);
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "Current sclk_g3d[G3D_BLK] = %dMhz", clkrate/1000000);
+
+ /* To be revised */
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "\nPossible settings : 533, 450, 400, 266, 160, 100Mhz");
+ if (ret < PAGE_SIZE - 1)
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "\n");
+ else
+ {
+ buf[PAGE_SIZE-2] = '\n';
+ buf[PAGE_SIZE-1] = '\0';
+ ret = PAGE_SIZE-1;
+ }
+
+ return ret;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static ssize_t set_clock(struct device *dev, struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct kbase_device *kbdev;
+ struct exynos_context *platform;
+ unsigned int tmp = 0;
+ unsigned int cmd = 0;
+ kbdev = dev_get_drvdata(dev);
+
+ if (!kbdev)
+ return -ENODEV;
+
+ platform = (struct exynos_context *) kbdev->platform_context;
+ if(!platform)
+ return -ENODEV;
+
+ if(!platform->sclk_g3d)
+ return -ENODEV;
+
+ if (sysfs_streq("533", buf)) {
+ cmd = 1;
+ kbase_platform_set_voltage( dev, 1250000 );
+ kbase_platform_dvfs_set_clock(kbdev, 533);
+ } else if (sysfs_streq("450", buf)) {
+ cmd = 1;
+ kbase_platform_set_voltage( dev, 1150000 );
+ kbase_platform_dvfs_set_clock(kbdev, 450);
+ } else if (sysfs_streq("400", buf)) {
+ cmd = 1;
+ kbase_platform_set_voltage( dev, 1100000 );
+ kbase_platform_dvfs_set_clock(kbdev, 400);
+ } else if (sysfs_streq("266", buf)) {
+ cmd = 1;
+ kbase_platform_set_voltage( dev, 937500);
+ kbase_platform_dvfs_set_clock(kbdev, 266);
+ } else if (sysfs_streq("160", buf)) {
+ cmd = 1;
+ kbase_platform_set_voltage( dev, 937500 );
+ kbase_platform_dvfs_set_clock(kbdev, 160);
+ } else if (sysfs_streq("100", buf)) {
+ cmd = 1;
+ kbase_platform_set_voltage( dev, 937500 );
+ kbase_platform_dvfs_set_clock(kbdev, 100);
+ } else {
+ dev_err(dev, "set_clock: invalid value\n");
+ return -ENOENT;
+ }
+
+ if(cmd == 1) {
+ /* Waiting for clock is stable */
+ do {
+ tmp = __raw_readl(/*EXYNOS5_CLKDIV_STAT_TOP0*/EXYNOS_CLKREG(0x10610));
+ } while (tmp & 0x1000000);
+ }
+ else if(cmd == 2) {
+ /* Do we need to check */
+ }
+
+ return count;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static ssize_t show_fbdev(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct kbase_device *kbdev;
+ ssize_t ret = 0;
+ int i;
+
+ kbdev = dev_get_drvdata(dev);
+
+ if (!kbdev)
+ return -ENODEV;
+
+ for(i = 0 ; i < num_registered_fb ; i++) {
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "fb[%d] xres=%d, yres=%d, addr=0x%lx\n", i, registered_fb[i]->var.xres, registered_fb[i]->var.yres, registered_fb[i]->fix.smem_start);
+ }
+
+ if (ret < PAGE_SIZE - 1)
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "\n");
+ else
+ {
+ buf[PAGE_SIZE-2] = '\n';
+ buf[PAGE_SIZE-1] = '\0';
+ ret = PAGE_SIZE-1;
+ }
+
+ return ret;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+typedef enum {
+ L1_I_tag_RAM = 0x00,
+ L1_I_data_RAM = 0x01,
+ L1_I_BTB_RAM = 0x02,
+ L1_I_GHB_RAM = 0x03,
+ L1_I_TLB_RAM = 0x04,
+ L1_I_indirect_predictor_RAM = 0x05,
+ L1_D_tag_RAM = 0x08,
+ L1_D_data_RAM = 0x09,
+ L1_D_load_TLB_array = 0x0A,
+ L1_D_store_TLB_array = 0x0B,
+ L2_tag_RAM = 0x10,
+ L2_data_RAM = 0x11,
+ L2_snoop_tag_RAM = 0x12,
+ L2_data_ECC_RAM = 0x13,
+ L2_dirty_RAM = 0x14,
+ L2_TLB_RAM = 0x18
+} RAMID_type;
+
+static inline void asm_ramindex_mrc(u32 *DL1Data0, u32 *DL1Data1, u32 *DL1Data2, u32 *DL1Data3)
+{
+ u32 val;
+
+ if(DL1Data0)
+ {
+ asm volatile("mrc p15, 0, %0, c15, c1, 0" : "=r" (val));
+ *DL1Data0 = val;
+ }
+ if(DL1Data1)
+ {
+ asm volatile("mrc p15, 0, %0, c15, c1, 1" : "=r" (val));
+ *DL1Data1 = val;
+ }
+ if(DL1Data2)
+ {
+ asm volatile("mrc p15, 0, %0, c15, c1, 2" : "=r" (val));
+ *DL1Data2 = val;
+ }
+ if(DL1Data3)
+ {
+ asm volatile("mrc p15, 0, %0, c15, c1, 3" : "=r" (val));
+ *DL1Data3 = val;
+ }
+}
+
+static inline void asm_ramindex_mcr(u32 val)
+{
+ asm volatile("mcr p15, 0, %0, c15, c4, 0" : : "r" (val));
+ asm volatile("dsb");
+ asm volatile("isb");
+}
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static void get_tlb_array(u32 val, u32 *DL1Data0, u32 *DL1Data1, u32 *DL1Data2, u32 *DL1Data3)
+{
+ asm_ramindex_mcr(val);
+ asm_ramindex_mrc(DL1Data0, DL1Data1, DL1Data2, DL1Data3);
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static RAMID_type ramindex = L1_D_load_TLB_array;
+static ssize_t show_dtlb(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct kbase_device *kbdev;
+ ssize_t ret = 0;
+ int entries, ways;
+ u32 DL1Data0 = 0, DL1Data1 = 0, DL1Data2 = 0, DL1Data3 = 0;
+
+ kbdev = dev_get_drvdata(dev);
+
+ if (!kbdev)
+ return -ENODEV;
+
+ /* L1-I tag RAM */
+ if(ramindex == L1_I_tag_RAM)
+ {
+ printk("Not implemented yet\n");
+ }
+ /* L1-I data RAM */
+ else if(ramindex == L1_I_data_RAM)
+ {
+ printk("Not implemented yet\n");
+ }
+ /* L1-I BTB RAM */
+ else if(ramindex == L1_I_BTB_RAM)
+ {
+ printk("Not implemented yet\n");
+ }
+ /* L1-I GHB RAM */
+ else if(ramindex == L1_I_GHB_RAM)
+ {
+ printk("Not implemented yet\n");
+ }
+ /* L1-I TLB RAM */
+ else if(ramindex == L1_I_TLB_RAM)
+ {
+ printk("L1-I TLB RAM\n");
+ for(entries = 0 ; entries < 32 ; entries++)
+ {
+ get_tlb_array((((u8)ramindex) << 24) + entries, &DL1Data0, &DL1Data1, &DL1Data2, NULL);
+ printk("entries[%d], DL1Data0=%08x, DL1Data1=%08x DL1Data2=%08x\n", entries, DL1Data0, DL1Data1 & 0xffff, 0x0);
+ }
+ }
+ /* L1-I indirect predictor RAM */
+ else if(ramindex == L1_I_indirect_predictor_RAM)
+ {
+ printk("Not implemented yet\n");
+ }
+ /* L1-D tag RAM */
+ else if(ramindex == L1_D_tag_RAM)
+ {
+ printk("Not implemented yet\n");
+ }
+ /* L1-D data RAM */
+ else if(ramindex == L1_D_data_RAM)
+ {
+ printk("Not implemented yet\n");
+ }
+ /* L1-D load TLB array */
+ else if(ramindex == L1_D_load_TLB_array)
+ {
+ printk("L1-D load TLB array\n");
+ for(entries = 0 ; entries < 32 ; entries++)
+ {
+ get_tlb_array((((u8)ramindex) << 24) + entries, &DL1Data0, &DL1Data1, &DL1Data2, &DL1Data3);
+ printk("entries[%d], DL1Data0=%08x, DL1Data1=%08x, DL1Data2=%08x, DL1Data3=%08x\n", entries, DL1Data0, DL1Data1, DL1Data2, DL1Data3 & 0x3f);
+ }
+ }
+ /* L1-D store TLB array */
+ else if(ramindex == L1_D_store_TLB_array)
+ {
+ printk("\nL1-D store TLB array\n");
+ for(entries = 0 ; entries < 32 ; entries++)
+ {
+ get_tlb_array((((u8)ramindex) << 24) + entries, &DL1Data0, &DL1Data1, &DL1Data2, &DL1Data3);
+ printk("entries[%d], DL1Data0=%08x, DL1Data1=%08x, DL1Data2=%08x, DL1Data3=%08x\n", entries, DL1Data0, DL1Data1, DL1Data2, DL1Data3 & 0x3f);
+ }
+ }
+ /* L2 tag RAM */
+ else if(ramindex == L2_tag_RAM)
+ {
+ printk("Not implemented yet\n");
+ }
+ /* L2 data RAM */
+ else if(ramindex == L2_data_RAM)
+ {
+ printk("Not implemented yet\n");
+ }
+ /* L2 snoop tag RAM */
+ else if(ramindex == L2_snoop_tag_RAM)
+ {
+ printk("Not implemented yet\n");
+ }
+ /* L2 data ECC RAM */
+ else if(ramindex == L2_data_ECC_RAM)
+ {
+ printk("Not implemented yet\n");
+ }
+ /* L2 dirty RAM */
+ else if(ramindex == L2_dirty_RAM)
+ {
+ printk("Not implemented yet\n");
+ }
+ /* L2 TLB array */
+ else if(ramindex == L2_TLB_RAM)
+ {
+ printk("\nL2 TLB array\n");
+ for(ways = 0 ; ways < 4 ; ways++)
+ {
+ for(entries = 0 ; entries < 512 ; entries++)
+ {
+ get_tlb_array((ramindex << 24) + (ways << 18) + entries, &DL1Data0, &DL1Data1, &DL1Data2, &DL1Data3);
+ printk("ways[%d]:entries[%d], DL1Data0=%08x, DL1Data1=%08x, DL1Data2=%08x, DL1Data3=%08x\n", ways, entries, DL1Data0, DL1Data1, DL1Data2, DL1Data3);
+ }
+ }
+ }
+
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "Succeeded...\n");
+
+ if (ret < PAGE_SIZE - 1)
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "\n");
+ else
+ {
+ buf[PAGE_SIZE-2] = '\n';
+ buf[PAGE_SIZE-1] = '\0';
+ ret = PAGE_SIZE-1;
+ }
+ return ret;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static ssize_t set_dtlb(struct device *dev, struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct kbase_device *kbdev;
+ kbdev = dev_get_drvdata(dev);
+
+ if (!kbdev)
+ return -ENODEV;
+
+ if (sysfs_streq("L1_I_tag_RAM", buf)) {
+ ramindex = L1_I_tag_RAM;
+ } else if (sysfs_streq("L1_I_data_RAM", buf)) {
+ ramindex = L1_I_data_RAM;
+ } else if (sysfs_streq("L1_I_BTB_RAM", buf)) {
+ ramindex = L1_I_BTB_RAM;
+ } else if (sysfs_streq("L1_I_GHB_RAM", buf)) {
+ ramindex = L1_I_GHB_RAM;
+ } else if (sysfs_streq("L1_I_TLB_RAM", buf)) {
+ ramindex = L1_I_TLB_RAM;
+ } else if (sysfs_streq("L1_I_indirect_predictor_RAM", buf)) {
+ ramindex = L1_I_indirect_predictor_RAM;
+ } else if (sysfs_streq("L1_D_tag_RAM", buf)) {
+ ramindex = L1_D_tag_RAM;
+ } else if (sysfs_streq("L1_D_data_RAM", buf)) {
+ ramindex = L1_D_data_RAM;
+ } else if (sysfs_streq("L1_D_load_TLB_array", buf)) {
+ ramindex = L1_D_load_TLB_array;
+ } else if (sysfs_streq("L1_D_store_TLB_array", buf)) {
+ ramindex = L1_D_store_TLB_array;
+ } else if (sysfs_streq("L2_tag_RAM", buf)) {
+ ramindex = L2_tag_RAM;
+ } else if (sysfs_streq("L2_data_RAM", buf)) {
+ ramindex = L2_data_RAM;
+ } else if (sysfs_streq("L2_snoop_tag_RAM", buf)) {
+ ramindex = L2_snoop_tag_RAM;
+ } else if (sysfs_streq("L2_data_ECC_RAM", buf)) {
+ ramindex = L2_data_ECC_RAM;
+ } else if (sysfs_streq("L2_dirty_RAM", buf)) {
+ ramindex = L2_dirty_RAM;
+ } else if (sysfs_streq("L2_TLB_RAM", buf)) {
+ ramindex = L2_TLB_RAM;
+ } else {
+ printk("Invalid value....\n\n");
+ printk("Available options are one of below\n");
+ printk("L1_I_tag_RAM, L1_I_data_RAM, L1_I_BTB_RAM\n");
+ printk("L1_I_GHB_RAM, L1_I_TLB_RAM, L1_I_indirect_predictor_RAM\n");
+ printk("L1_D_tag_RAM, L1_D_data_RAM, L1_D_load_TLB_array, L1_D_store_TLB_array\n");
+ printk("L2_tag_RAM, L2_data_RAM, L2_snoop_tag_RAM, L2_data_ECC_RAM\n");
+ printk("L2_dirty_RAM, L2_TLB_RAM\n");
+ }
+
+ return count;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static ssize_t show_vol(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct kbase_device *kbdev;
+ ssize_t ret = 0;
+ int vol;
+
+ kbdev = dev_get_drvdata(dev);
+
+ if (!kbdev)
+ return -ENODEV;
+
+ kbase_platform_get_voltage(dev, &vol);
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "Current operating voltage for vithar = %d", vol);
+
+ if (ret < PAGE_SIZE - 1)
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "\n");
+ else
+ {
+ buf[PAGE_SIZE-2] = '\n';
+ buf[PAGE_SIZE-1] = '\0';
+ ret = PAGE_SIZE-1;
+ }
+
+ return ret;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static int get_clkout_cmu_top(int *val)
+{
+ *val = __raw_readl(/*EXYNOS5_CLKOUT_CMU_TOP*/EXYNOS_CLKREG(0x10A00));
+ if((*val & 0x1f) == 0xB) /* CLKOUT is ACLK_400 in CLKOUT_CMU_TOP */
+ return 1;
+ else
+ return 0;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static void set_clkout_for_3d(void)
+{
+ int tmp;
+
+ tmp = 0x0;
+ tmp |= 0x1000B; /* ACLK_400 selected */
+ tmp |= 9 << 8; /* divided by (9 + 1) */
+ __raw_writel(tmp, /*EXYNOS5_CLKOUT_CMU_TOP*/EXYNOS_CLKREG(0x10A00));
+
+#ifdef PMU_XCLKOUT_SET
+ exynos5_pmu_xclkout_set(1, XCLKOUT_CMU_TOP);
+#else /* PMU_XCLKOUT_SET */
+ tmp = 0x0;
+ tmp |= 7 << 8; /* CLKOUT_CMU_TOP selected */
+ __raw_writel(tmp, /*S5P_PMU_DEBUG*/S5P_PMUREG(0x0A00));
+#endif /* PMU_XCLKOUT_SET */
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static ssize_t show_clkout(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct kbase_device *kbdev;
+ ssize_t ret = 0;
+ int val;
+
+ kbdev = dev_get_drvdata(dev);
+
+ if (!kbdev)
+ return -ENODEV;
+
+ if(get_clkout_cmu_top(&val))
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "Current CLKOUT is g3d divided by 10, CLKOUT_CMU_TOP=0x%x", val);
+ else
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "Current CLKOUT is not g3d, CLKOUT_CMU_TOP=0x%x", val);
+
+ if (ret < PAGE_SIZE - 1)
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "\n");
+ else
+ {
+ buf[PAGE_SIZE-2] = '\n';
+ buf[PAGE_SIZE-1] = '\0';
+ ret = PAGE_SIZE-1;
+ }
+
+ return ret;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static ssize_t set_clkout(struct device *dev, struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct kbase_device *kbdev;
+ kbdev = dev_get_drvdata(dev);
+
+ if (!kbdev)
+ return -ENODEV;
+
+ if (sysfs_streq("3d", buf)) {
+ set_clkout_for_3d();
+ } else {
+ printk("invalid val (only 3d is accepted\n");
+ }
+
+ return count;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static ssize_t show_dvfs(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct kbase_device *kbdev;
+ ssize_t ret = 0;
+
+ kbdev = dev_get_drvdata(dev);
+
+ if (!kbdev)
+ return -ENODEV;
+
+#ifdef CONFIG_VITHAR_DVFS
+ if(kbdev->pm.metrics.timer.active == MALI_FALSE )
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "G3D DVFS is off");
+ else
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "G3D DVFS is on");
+#else /* CONFIG_VITHAR_DVFS */
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "G3D DVFS is disabled");
+#endif /* CONFIG_VITHAR_DVFS */
+
+ if (ret < PAGE_SIZE - 1)
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "\n");
+ else
+ {
+ buf[PAGE_SIZE-2] = '\n';
+ buf[PAGE_SIZE-1] = '\0';
+ ret = PAGE_SIZE-1;
+ }
+
+ return ret;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static ssize_t set_dvfs(struct device *dev, struct device_attribute *attr, const char *buf, size_t count)
+{
+#ifdef CONFIG_VITHAR_DVFS
+ osk_error ret;
+ int vol;
+#endif /* CONFIG_VITHAR_DVFS */
+ struct kbase_device *kbdev;
+ kbdev = dev_get_drvdata(dev);
+
+ if (!kbdev)
+ return -ENODEV;
+
+#ifdef CONFIG_VITHAR_DVFS
+ if (sysfs_streq("off", buf)) {
+ if(kbdev->pm.metrics.timer.active == MALI_FALSE )
+ return count;
+
+ osk_spinlock_irq_lock(&kbdev->pm.metrics.lock);
+ kbdev->pm.metrics.timer_active = MALI_FALSE;
+ osk_spinlock_irq_unlock(&kbdev->pm.metrics.lock);
+
+ osk_timer_stop(&kbdev->pm.metrics.timer);
+
+ kbase_platform_get_default_voltage(dev, &vol);
+ if(vol != 0)
+ kbase_platform_set_voltage(dev, vol);
+ kbase_platform_dvfs_set_clock(kbdev,VITHAR_DEFAULT_CLOCK / 1000000);
+ } else if (sysfs_streq("on", buf)) {
+ if(kbdev->pm.metrics.timer_active == MALI_TRUE )
+ return count;
+
+ osk_spinlock_irq_lock(&kbdev->pm.metrics.lock);
+ kbdev->pm.metrics.timer_active = MALI_TRUE;
+ osk_spinlock_irq_unlock(&kbdev->pm.metrics.lock);
+
+ ret = osk_timer_start(&kbdev->pm.metrics.timer, KBASE_PM_DVFS_FREQUENCY);
+ if (ret != OSK_ERR_NONE)
+ {
+ printk("osk_timer_start failed\n");
+ }
+ } else {
+ printk("invalid val -only [on, off] is accepted\n");
+ }
+#else /* CONFIG_VITHAR_DVFS */
+ printk("G3D DVFS is disabled\n");
+#endif /* CONFIG_VITHAR_DVFS */
+
+ return count;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#define KBASE_PM_DVFS_FREQUENCY 100
+
+#define MALI_DVFS_DEBUG 0
+#define MALI_DVFS_START_MAX_STEP 1
+#define MALI_DVFS_STEP 6
+#define MALI_DVFS_KEEP_STAY_CNT 10
+#define CONFIG_VITHAR_FREQ_LOCK
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static ssize_t show_lock_dvfs(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct kbase_device *kbdev;
+ ssize_t ret = 0;
+#ifdef CONFIG_VITHAR_DVFS
+ unsigned int locked_level = -1;
+#endif /* CONFIG_VITHAR_DVFS */
+
+ kbdev = dev_get_drvdata(dev);
+
+ if (!kbdev)
+ return -ENODEV;
+
+#ifdef CONFIG_VITHAR_DVFS
+ locked_level = mali_get_dvfs_upper_locked_freq();
+ if( locked_level > 0 )
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "Current Upper Lock Level = %dMhz", locked_level );
+ else
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "Unset the Upper Lock Level");
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "\nPossible settings : 450, 400, 266, 160, 100, If you want to unlock : 533");
+
+#else /* CONFIG_VITHAR_DVFS */
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "G3D DVFS is disabled. You can not setting the Upper Lock level.");
+#endif /* CONFIG_VITHAR_DVFS */
+
+ if (ret < PAGE_SIZE - 1)
+ ret += snprintf(buf+ret, PAGE_SIZE-ret, "\n");
+ else
+ {
+ buf[PAGE_SIZE-2] = '\n';
+ buf[PAGE_SIZE-1] = '\0';
+ ret = PAGE_SIZE-1;
+ }
+
+ return ret;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+static ssize_t set_lock_dvfs(struct device *dev, struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct kbase_device *kbdev;
+ kbdev = dev_get_drvdata(dev);
+
+ if (!kbdev)
+ return -ENODEV;
+
+#ifdef CONFIG_VITHAR_DVFS
+#if (MALI_DVFS_STEP == 6)
+ if (sysfs_streq("533", buf)) {
+ mali_dvfs_freq_unlock();
+ } else if (sysfs_streq("450", buf)) {
+ mali_dvfs_freq_lock(4);
+ } else if (sysfs_streq("400", buf)) {
+ mali_dvfs_freq_lock(3);
+ } else if (sysfs_streq("266", buf)) {
+ mali_dvfs_freq_lock(2);
+ } else if (sysfs_streq("160", buf)) {
+ mali_dvfs_freq_lock(1);
+ } else if (sysfs_streq("100", buf)) {
+ mali_dvfs_freq_lock(0);
+ } else {
+ dev_err(dev, "set_clock: invalid value\n");
+ dev_err(dev, "Possible settings : 450, 400, 266, 160, 100, If you want to unlock : 533\n");
+ return -ENOENT;
+ }
+#elif (MALI_DVFS_STEP == 2)
+ if (sysfs_streq("533", buf)) {
+ mali_dvfs_freq_unlock();
+ } else if (sysfs_streq("266", buf)) {
+ mali_dvfs_freq_lock(0);
+ } else {
+ dev_err(dev, "set_clock: invalid value\n");
+ dev_err(dev, "Possible settings : 450, 400, 266, 160, 100, If you want to unlock : 533\n");
+ return -ENOENT;
+ }
+#endif /* MALI_DVFS_STEP */
+#else /* CONFIG_VITHAR_DVFS */
+ printk("G3D DVFS is disabled. You can not setting the Upper Lock level.\n");
+#endif /* CONFIG_VITHAR_DVFS */
+
+ return count;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+/** The sysfs file @c clock, fbdev.
+ *
+ * This is used for obtaining information about the vithar operating clock & framebuffer address,
+ */
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+DEVICE_ATTR(clock, S_IRUGO|S_IWUSR, show_clock, set_clock);
+DEVICE_ATTR(fbdev, S_IRUGO, show_fbdev, NULL);
+DEVICE_ATTR(dtlb, S_IRUGO|S_IWUSR, show_dtlb, set_dtlb);
+DEVICE_ATTR(vol, S_IRUGO|S_IWUSR, show_vol, NULL);
+DEVICE_ATTR(clkout, S_IRUGO|S_IWUSR, show_clkout, set_clkout);
+DEVICE_ATTR(dvfs, S_IRUGO|S_IWUSR, show_dvfs, set_dvfs);
+DEVICE_ATTR(dvfs_lock, S_IRUGO|S_IWUSR, show_lock_dvfs, set_lock_dvfs);
+
+static int kbase_platform_create_sysfs_file(struct device *dev)
+{
+ if (device_create_file(dev, &dev_attr_clock))
+ {
+ dev_err(dev, "Couldn't create sysfs file [clock]\n");
+ goto out;
+ }
+
+ if (device_create_file(dev, &dev_attr_fbdev))
+ {
+ dev_err(dev, "Couldn't create sysfs file [fbdev]\n");
+ goto out;
+ }
+
+ if (device_create_file(dev, &dev_attr_dtlb))
+ {
+ dev_err(dev, "Couldn't create sysfs file [dtlb]\n");
+ goto out;
+ }
+
+ if (device_create_file(dev, &dev_attr_vol))
+ {
+ dev_err(dev, "Couldn't create sysfs file [vol]\n");
+ goto out;
+ }
+
+ if (device_create_file(dev, &dev_attr_clkout))
+ {
+ dev_err(dev, "Couldn't create sysfs file [clkout]\n");
+ goto out;
+ }
+
+ if (device_create_file(dev, &dev_attr_dvfs))
+ {
+ dev_err(dev, "Couldn't create sysfs file [dvfs]\n");
+ goto out;
+ }
+
+ if (device_create_file(dev, &dev_attr_dvfs_lock))
+ {
+ dev_err(dev, "Couldn't create sysfs file [dvfs_lock]\n");
+ goto out;
+ }
+ return 0;
+out:
+ return -ENOENT;
+}
+
+void kbase_platform_remove_sysfs_file(struct device *dev)
+{
+ device_remove_file(dev, &dev_attr_clock);
+ device_remove_file(dev, &dev_attr_fbdev);
+ device_remove_file(dev, &dev_attr_dtlb);
+ device_remove_file(dev, &dev_attr_vol);
+ device_remove_file(dev, &dev_attr_clkout);
+ device_remove_file(dev, &dev_attr_dvfs);
+ device_remove_file(dev, &dev_attr_dvfs_lock);
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#include "osk/include/mali_osk_lock_order.h"
+
+#ifdef CONFIG_PM_RUNTIME
+static void kbase_platform_runtime_term(struct kbase_device *kbdev)
+{
+ pm_runtime_disable(kbdev->osdev.dev);
+}
+#endif /* CONFIG_PM_RUNTIME */
+
+#ifdef CONFIG_PM_RUNTIME
+extern void pm_runtime_init(struct device *dev);
+
+static mali_error kbase_platform_runtime_init(struct kbase_device *kbdev)
+{
+ pm_runtime_init(kbdev->osdev.dev);
+ pm_suspend_ignore_children(kbdev->osdev.dev, true);
+ pm_runtime_enable(kbdev->osdev.dev);
+ return MALI_ERROR_NONE;
+}
+#endif /* CONFIG_PM_RUNTIME */
+
+
+mali_error kbase_platform_init(kbase_device *kbdev)
+{
+ struct exynos_context *platform;
+
+ platform = osk_malloc(sizeof(struct exynos_context));
+
+ if(NULL == platform)
+ {
+ return MALI_ERROR_OUT_OF_MEMORY;
+ }
+
+ kbdev->platform_context = (void *) platform;
+
+ platform->cmu_pmu_status = 0;
+ spin_lock_init(&platform->cmu_pmu_lock);
+
+ if(kbase_platform_power_clock_init(kbdev))
+ {
+ goto clock_init_fail;
+ }
+
+#ifdef CONFIG_REGULATOR
+ if(kbase_platform_regulator_init())
+ {
+ goto regulator_init_fail;
+ }
+#endif /* CONFIG_REGULATOR */
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+ if(kbase_platform_create_sysfs_file(dev))
+ {
+ return create_sysfs_file_fail;
+ }
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#ifdef CONFIG_VITHAR_DVFS
+ kbase_platform_dvfs_init(kbdev);
+#endif /* CONFIG_VITHAR_DVFS */
+
+ /* Enable power */
+ kbase_platform_cmu_pmu_control(kbdev, 1);
+ return MALI_ERROR_NONE;
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+create_sysfs_file_fail:
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+#ifdef CONFIG_REGULATOR
+ kbase_platform_regulator_disable();
+#endif /* CONFIG_REGULATOR */
+regulator_init_fail:
+clock_init_fail:
+ osk_free(platform);
+
+ return MALI_ERROR_FUNCTION_FAILED;
+}
+
+void kbase_platform_term(kbase_device *kbdev)
+{
+ struct exynos_context *platform;
+
+ platform = (struct exynos_context *) kbdev->platform_context;
+
+#ifdef CONFIG_VITHAR_DVFS
+ kbase_platform_dvfs_term();
+#endif /* CONFIG_VITHAR_DVFS */
+
+ /* Disable power */
+ kbase_platform_cmu_pmu_control(kbdev, 0);
+#ifdef CONFIG_REGULATOR
+ kbase_platform_regulator_disable();
+#endif /* CONFIG_REGULATOR */
+ osk_free(kbdev->platform_context);
+ kbdev->platform_context = 0;
+ return;
+}
+
+#ifdef CONFIG_REGULATOR
+static struct regulator *g3d_regulator=NULL;
+#ifdef CONFIG_VITHAR_HWVER_R0P0
+static int mali_gpu_vol = 1250000; /* 1.25V @ 533 MHz */
+#else
+static int mali_gpu_vol = 1050000; /* 1.05V @ 266 MHz */
+#endif /* CONFIG_VITHAR_HWVER_R0P0 */
+#endif /* CONFIG_REGULATOR */
+
+#ifdef CONFIG_VITHAR_DVFS
+typedef struct _mali_dvfs_info{
+ unsigned int voltage;
+ unsigned int clock;
+ int min_threshold;
+ int max_threshold;
+}mali_dvfs_info;
+
+typedef struct _mali_dvfs_status_type{
+ kbase_device *kbdev;
+ int step;
+ int utilisation;
+ int keepcnt;
+ uint noutilcnt;
+#ifdef CONFIG_VITHAR_FREQ_LOCK
+ int upper_lock;
+ int under_lock;
+#endif /* CONFIG_VITHAR_FREQ_LOCK */
+}mali_dvfs_status;
+
+static struct workqueue_struct *mali_dvfs_wq = 0;
+int mali_dvfs_control=0;
+osk_spinlock mali_dvfs_spinlock;
+
+
+/*dvfs status*/
+static mali_dvfs_status mali_dvfs_status_current;
+static const mali_dvfs_info mali_dvfs_infotbl[MALI_DVFS_STEP]=
+{
+#if (MALI_DVFS_STEP == 6)
+ {937500/*750000*/, 100, 0, 40},
+ {937500/*81250*/, 160, 30, 60},
+ {937500, 266, 50, 70},
+ {1100000, 400, 60, 80},
+ {1150000, 450, 70, 90},
+ {1250000, 533, 80, 100}
+#elif (MALI_DVFS_STEP == 2)
+ {937500, 266, 0, 55},
+ {1250000, 533, 45, 100}
+#else /* MALI_DVFS_STEP */
+#error no table
+#endif /* MALI_DVFS_STEP */
+};
+
+static void kbase_platform_dvfs_set_vol(unsigned int vol)
+{
+ static int _vol = -1;
+
+ if (_vol == vol)
+ return;
+
+
+ switch(vol)
+ {
+ case 1250000:
+ case 1150000:
+ case 1100000:
+ case 1000000:
+ case 937500:
+ case 812500:
+ case 750000:
+ kbase_platform_set_voltage(NULL, vol);
+ break;
+ default:
+ return;
+ }
+
+ _vol = vol;
+
+#if MALI_DVFS_DEBUG
+ printk("dvfs_set_vol %dmV\n", vol);
+#endif /* MALI_DVFS_DEBUG */
+ return;
+}
+
+#ifdef CONFIG_VITHAR_FREQ_LOCK
+int mali_get_dvfs_upper_locked_freq(void)
+{
+ unsigned int locked_level = -1;
+
+ osk_spinlock_lock(&mali_dvfs_spinlock);
+ locked_level = mali_dvfs_infotbl[mali_dvfs_status_current.upper_lock].clock;
+ osk_spinlock_unlock(&mali_dvfs_spinlock);
+
+ return locked_level;
+}
+
+int mali_get_dvfs_under_locked_freq(void)
+{
+ unsigned int locked_level = -1;
+
+ osk_spinlock_lock(&mali_dvfs_spinlock);
+ locked_level = mali_dvfs_infotbl[mali_dvfs_status_current.under_lock].clock;
+ osk_spinlock_unlock(&mali_dvfs_spinlock);
+
+ return locked_level;
+}
+
+
+int mali_dvfs_freq_lock(int level)
+{
+ osk_spinlock_lock(&mali_dvfs_spinlock);
+ mali_dvfs_status_current.upper_lock = level;
+ osk_spinlock_unlock(&mali_dvfs_spinlock);
+ return 0;
+}
+void mali_dvfs_freq_unlock(void)
+{
+ osk_spinlock_lock(&mali_dvfs_spinlock);
+ mali_dvfs_status_current.upper_lock = -1;
+ osk_spinlock_unlock(&mali_dvfs_spinlock);
+}
+
+int mali_dvfs_freq_under_lock(int level)
+{
+ osk_spinlock_lock(&mali_dvfs_spinlock);
+ mali_dvfs_status_current.under_lock = level;
+ osk_spinlock_unlock(&mali_dvfs_spinlock);
+ return 0;
+}
+void mali_dvfs_freq_under_unlock(void)
+{
+ osk_spinlock_lock(&mali_dvfs_spinlock);
+ mali_dvfs_status_current.under_lock = -1;
+ osk_spinlock_unlock(&mali_dvfs_spinlock);
+}
+#endif /* CONFIG_VITHAR_FREQ_LOCK */
+
+void kbase_platform_dvfs_set_level(int level)
+{
+ static int level_prev=-1;
+
+ if (level == level_prev)
+ return;
+
+ if (WARN_ON(level >= MALI_DVFS_STEP))
+ panic("invalid level");
+
+ if (level > level_prev) {
+ kbase_platform_dvfs_set_vol(mali_dvfs_infotbl[level].voltage);
+ kbase_platform_dvfs_set_clock(mali_dvfs_status_current.kbdev, mali_dvfs_infotbl[level].clock);
+ }else{
+ kbase_platform_dvfs_set_clock(mali_dvfs_status_current.kbdev, mali_dvfs_infotbl[level].clock);
+ kbase_platform_dvfs_set_vol(mali_dvfs_infotbl[level].voltage);
+ }
+ level_prev = level;
+}
+
+static void mali_dvfs_event_proc(struct work_struct *w)
+{
+ mali_dvfs_status dvfs_status;
+
+ osk_spinlock_lock(&mali_dvfs_spinlock);
+ dvfs_status = mali_dvfs_status_current;
+ osk_spinlock_unlock(&mali_dvfs_spinlock);
+
+#if MALI_DVFS_START_MAX_STEP
+ /*If no input is for longtime, first step will be max step. */
+ if (dvfs_status.utilisation > 10 && dvfs_status.noutilcnt > 20) {
+ dvfs_status.step=MALI_DVFS_STEP-2;
+ dvfs_status.utilisation = 100;
+ }
+#endif /* MALI_DVFS_START_MAX_STEP */
+
+ if (dvfs_status.utilisation > mali_dvfs_infotbl[dvfs_status.step].max_threshold)
+ {
+ OSK_ASSERT(dvfs_status.step < MALI_DVFS_STEP);
+ dvfs_status.step++;
+ dvfs_status.keepcnt=0;
+ }else if ((dvfs_status.step>0) &&
+ (dvfs_status.utilisation < mali_dvfs_infotbl[dvfs_status.step].min_threshold)) {
+ dvfs_status.keepcnt++;
+ if (dvfs_status.keepcnt > MALI_DVFS_KEEP_STAY_CNT)
+ {
+ OSK_ASSERT(dvfs_status.step > 0);
+ dvfs_status.step--;
+ dvfs_status.keepcnt=0;
+ }
+ }else{
+ dvfs_status.keepcnt=0;
+ }
+
+#ifdef CONFIG_VITHAR_FREQ_LOCK
+ if ((dvfs_status.upper_lock > 0)&&(dvfs_status.step > dvfs_status.upper_lock)) {
+ dvfs_status.step = dvfs_status.upper_lock;
+ if ((dvfs_status.under_lock > 0)&&(dvfs_status.under_lock > dvfs_status.upper_lock)) {
+ dvfs_status.under_lock = dvfs_status.upper_lock;
+ }
+ }
+ if (dvfs_status.under_lock > 0) {
+ if (dvfs_status.step < dvfs_status.under_lock)
+ dvfs_status.step = dvfs_status.under_lock;
+ }
+#endif /* CONFIG_VITHAR_FREQ_LOCK */
+
+ kbase_platform_dvfs_set_level(dvfs_status.step);
+
+#if MALI_DVFS_START_MAX_STEP
+ if (dvfs_status.utilisation == 0) {
+ dvfs_status.noutilcnt++;
+ } else {
+ dvfs_status.noutilcnt=0;
+ }
+#endif /* MALI_DVFS_START_MAX_STEP */
+
+#if MALI_DVFS_DEBUG
+ printk("[mali_dvfs] utilisation: %d step: %d[%d,%d] cnt: %d\n",
+ dvfs_status.utilisation, dvfs_status.step,
+ mali_dvfs_infotbl[dvfs_status.step].min_threshold,
+ mali_dvfs_infotbl[dvfs_status.step].max_threshold, dvfs_status.keepcnt);
+#endif /* MALI_DVFS_DEBUG */
+
+ osk_spinlock_lock(&mali_dvfs_spinlock);
+ mali_dvfs_status_current=dvfs_status;
+ osk_spinlock_unlock(&mali_dvfs_spinlock);
+
+}
+
+static DECLARE_WORK(mali_dvfs_work, mali_dvfs_event_proc);
+
+int kbase_platform_dvfs_event(struct kbase_device *kbdev, u32 utilisation)
+{
+ osk_spinlock_lock(&mali_dvfs_spinlock);
+ mali_dvfs_status_current.utilisation = utilisation;
+ osk_spinlock_unlock(&mali_dvfs_spinlock);
+ queue_work_on(0, mali_dvfs_wq, &mali_dvfs_work);
+
+ /*add error handle here*/
+ return MALI_TRUE;
+}
+
+int kbase_platform_dvfs_get_control_status(void)
+{
+ return mali_dvfs_control;
+}
+
+int kbase_platform_dvfs_init(kbase_device *kbdev)
+{
+ /*default status
+ add here with the right function to get initilization value.
+ */
+ if (!mali_dvfs_wq)
+ mali_dvfs_wq = create_singlethread_workqueue("mali_dvfs");
+
+ osk_spinlock_init(&mali_dvfs_spinlock,OSK_LOCK_ORDER_PM_METRICS);
+
+ /*add a error handling here*/
+ osk_spinlock_lock(&mali_dvfs_spinlock);
+ mali_dvfs_status_current.kbdev = kbdev;
+ mali_dvfs_status_current.utilisation = 100;
+ mali_dvfs_status_current.step = MALI_DVFS_STEP-1;
+#ifdef CONFIG_VITHAR_FREQ_LOCK
+ mali_dvfs_status_current.upper_lock = -1;
+ mali_dvfs_status_current.under_lock = -1;
+#endif /* CONFIG_VITHAR_FREQ_LOCK */
+ osk_spinlock_unlock(&mali_dvfs_spinlock);
+
+ return MALI_TRUE;
+}
+
+void kbase_platform_dvfs_term(void)
+{
+ if (mali_dvfs_wq)
+ destroy_workqueue(mali_dvfs_wq);
+
+ mali_dvfs_wq = NULL;
+}
+#endif /* CONFIG_VITHAR_DVFS */
+
+int kbase_platform_regulator_init(void)
+{
+#ifdef CONFIG_REGULATOR
+ g3d_regulator = regulator_get(NULL, "vdd_g3d");
+ if(IS_ERR(g3d_regulator))
+ {
+ printk("[kbase_platform_regulator_init] failed to get vithar regulator\n");
+ return -1;
+ }
+
+ if(regulator_enable(g3d_regulator) != 0)
+ {
+ printk("[kbase_platform_regulator_init] failed to enable vithar regulator\n");
+ return -1;
+ }
+
+ if(regulator_set_voltage(g3d_regulator, mali_gpu_vol, mali_gpu_vol) != 0)
+ {
+ printk("[kbase_platform_regulator_init] failed to set vithar operating voltage [%d]\n", mali_gpu_vol);
+ return -1;
+ }
+#endif /* CONFIG_REGULATOR */
+
+ return 0;
+}
+
+int kbase_platform_regulator_disable(void)
+{
+#ifdef CONFIG_REGULATOR
+ if(!g3d_regulator)
+ {
+ printk("[kbase_platform_regulator_disable] g3d_regulator is not initialized\n");
+ return -1;
+ }
+
+ if(regulator_disable(g3d_regulator) != 0)
+ {
+ printk("[kbase_platform_regulator_disable] failed to disable g3d regulator\n");
+ return -1;
+ }
+#endif /* CONFIG_REGULATOR */
+ return 0;
+}
+
+int kbase_platform_regulator_enable(struct device *dev)
+{
+#ifdef CONFIG_REGULATOR
+ if(!g3d_regulator)
+ {
+ printk("[kbase_platform_regulator_enable] g3d_regulator is not initialized\n");
+ return -1;
+ }
+
+ if(regulator_enable(g3d_regulator) != 0)
+ {
+ printk("[kbase_platform_regulator_enable] failed to enable g3d regulator\n");
+ return -1;
+ }
+#endif /* CONFIG_REGULATOR */
+ return 0;
+}
+
+int kbase_platform_get_default_voltage(struct device *dev, int *vol)
+{
+#ifdef CONFIG_REGULATOR
+ *vol = mali_gpu_vol;
+#else /* CONFIG_REGULATOR */
+ *vol = 0;
+#endif /* CONFIG_REGULATOR */
+ return 0;
+}
+
+#ifdef CONFIG_VITHAR_DEBUG_SYS
+int kbase_platform_get_voltage(struct device *dev, int *vol)
+{
+#ifdef CONFIG_REGULATOR
+ if(!g3d_regulator)
+ {
+ printk("[kbase_platform_get_voltage] g3d_regulator is not initialized\n");
+ return -1;
+ }
+
+ *vol = regulator_get_voltage(g3d_regulator);
+#else /* CONFIG_REGULATOR */
+ *vol = 0;
+#endif /* CONFIG_REGULATOR */
+ return 0;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+#if defined CONFIG_VITHAR_DEBUG_SYS || defined CONFIG_VITHAR_DVFS
+int kbase_platform_set_voltage(struct device *dev, int vol)
+{
+#ifdef CONFIG_REGULATOR
+ if(!g3d_regulator)
+ {
+ printk("[kbase_platform_set_voltage] g3d_regulator is not initialized\n");
+ return -1;
+ }
+
+ if(regulator_set_voltage(g3d_regulator, vol, vol) != 0)
+ {
+ printk("[kbase_platform_set_voltage] failed to set voltage\n");
+ return -1;
+ }
+#endif /* CONFIG_REGULATOR */
+ return 0;
+}
+#endif /* CONFIG_VITHAR_DEBUG_SYS */
+
+void kbase_platform_dvfs_set_clock(kbase_device *kbdev, int freq)
+{
+ static struct clk * mout_gpll = NULL;
+ static struct clk * fin_gpll = NULL;
+ static struct clk * fout_gpll = NULL;
+ static int _freq = -1;
+ static unsigned long gpll_rate_prev = 0;
+ unsigned long gpll_rate = 0, aclk_400_rate = 0;
+ unsigned long tmp = 0;
+ struct exynos_context *platform;
+
+
+ if (!kbdev)
+ panic("oops");
+
+ platform = (struct exynos_context *) kbdev->platform_context;
+ if(NULL == platform)
+ {
+ panic("oops");
+ }
+
+ if (mout_gpll==NULL) {
+ mout_gpll = clk_get(kbdev->osdev.dev, "mout_gpll");
+ fin_gpll = clk_get(kbdev->osdev.dev, "ext_xtal");
+ fout_gpll = clk_get(kbdev->osdev.dev, "fout_gpll");
+ if(IS_ERR(mout_gpll) || IS_ERR(fin_gpll) || IS_ERR(fout_gpll))
+ panic("clk_get ERROR");
+ }
+
+ if(platform->sclk_g3d == 0)
+ return;
+
+ if (freq == _freq)
+ return;
+
+ switch(freq)
+ {
+ case 533:
+ gpll_rate = 533000000;
+ aclk_400_rate = 533000000;
+ break;
+ case 450:
+ gpll_rate = 450000000;
+ aclk_400_rate = 450000000;
+ break;
+ case 400:
+ gpll_rate = 800000000;
+ aclk_400_rate = 400000000;
+ break;
+ case 266:
+ gpll_rate = 800000000;
+ aclk_400_rate = 267000000;
+ break;
+ case 160:
+ gpll_rate = 800000000;
+ aclk_400_rate = 160000000;
+ break;
+ case 100:
+ gpll_rate = 800000000;
+ aclk_400_rate = 100000000;
+ break;
+ default:
+ return;
+ }
+
+ /* if changed the GPLL rate, set rate for GPLL and wait for lock time */
+ if( gpll_rate != gpll_rate_prev) {
+ /*for stable clock input.*/
+ clk_set_rate(platform->sclk_g3d, 100000000);
+ clk_set_parent(mout_gpll, fin_gpll);
+
+ /*change gpll*/
+ clk_set_rate( fout_gpll, gpll_rate );
+
+ /*restore parent*/
+ clk_set_parent(mout_gpll, fout_gpll);
+ gpll_rate_prev = gpll_rate;
+ }
+
+ _freq = freq;
+ clk_set_rate(platform->sclk_g3d, aclk_400_rate);
+
+ /* Waiting for clock is stable */
+ do {
+ tmp = __raw_readl(/*EXYNOS5_CLKDIV_STAT_TOP0*/EXYNOS_CLKREG(0x10610));
+ } while (tmp & 0x1000000);
+#ifdef CONFIG_VITHAR_DVFS
+#if MALI_DVFS_DEBUG
+ printk("aclk400 %u[%d]\n", (unsigned int)clk_get_rate(platform->sclk_g3d),mali_dvfs_status_current.utilisation);
+ printk("dvfs_set_clock GPLL : %lu, ACLK_400 : %luMhz\n", gpll_rate, aclk_400_rate );
+#endif /* MALI_DVFS_DEBUG */
+#endif /* CONFIG_VITHAR_DVFS */
+ return;
+}
+
+/**
+ * Exynos5 alternative dvfs_callback imlpementation.
+ * instead of:
+ * action = kbase_pm_get_dvfs_action(kbdev);
+ * use this:
+ * kbase_platform_dvfs_event(kbdev, kbase_pm_get_dvfs_utilisation(kbdev));
+ */
+
+#ifdef CONFIG_VITHAR_DVFS
+int kbase_pm_get_dvfs_utilisation(kbase_device *kbdev)
+{
+ int utilisation=0;
+ osk_ticks now = osk_time_now();
+
+ OSK_ASSERT(kbdev != NULL);
+
+ osk_spinlock_irq_lock(&kbdev->pm.metrics.lock);
+
+ if (kbdev->pm.metrics.gpu_active)
+ {
+ kbdev->pm.metrics.time_busy += osk_time_elapsed(kbdev->pm.metrics.time_period_start, now);
+ kbdev->pm.metrics.time_period_start = now;
+ }
+ else
+ {
+ kbdev->pm.metrics.time_idle += osk_time_elapsed(kbdev->pm.metrics.time_period_start, now);
+ kbdev->pm.metrics.time_period_start = now;
+ }
+
+ if (kbdev->pm.metrics.time_idle + kbdev->pm.metrics.time_busy == 0)
+ {
+ /* No data - so we return NOP */
+ goto out;
+ }
+
+ utilisation = (100*kbdev->pm.metrics.time_busy) / (kbdev->pm.metrics.time_idle + kbdev->pm.metrics.time_busy);
+
+out:
+
+ kbdev->pm.metrics.time_idle = 0;
+ kbdev->pm.metrics.time_busy = 0;
+
+ osk_spinlock_irq_unlock(&kbdev->pm.metrics.lock);
+
+ return utilisation;
+}
+#endif
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#include <linux/ioport.h>
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_kbase_defs.h>
+#include <kbase/src/linux/mali_kbase_config_linux.h>
+#include <ump/ump_common.h>
+
+#include "mali_kbase_cpu_vexpress.h"
+
+/* Versatile Express (VE) configuration defaults shared between config_attributes[]
+ * and config_attributes_hw_issue_8408[]. Settings are not shared for
+ * JS_HARD_STOP_TICKS_SS and JS_RESET_TICKS_SS.
+ */
+#define KBASE_VE_MEMORY_PER_PROCESS_LIMIT 512 * 1024 * 1024UL /* 512MB */
+#define KBASE_VE_UMP_DEVICE UMP_DEVICE_Z_SHIFT
+#define KBASE_VE_MEMORY_OS_SHARED_MAX 768 * 1024 * 1024UL /* 768MB */
+#define KBASE_VE_MEMORY_OS_SHARED_PERF_GPU KBASE_MEM_PERF_SLOW
+#define KBASE_VE_GPU_FREQ_KHZ_MAX 5000
+#define KBASE_VE_GPU_FREQ_KHZ_MIN 5000
+
+#define KBASE_VE_JS_SCHEDULING_TICK_NS_DEBUG 15000000u /* 15ms, an agressive tick for testing purposes. This will reduce performance significantly */
+#define KBASE_VE_JS_SOFT_STOP_TICKS_DEBUG 1 /* between 15ms and 30ms before soft-stop a job */
+#define KBASE_VE_JS_HARD_STOP_TICKS_SS_DEBUG 333 /* 5s before hard-stop */
+#define KBASE_VE_JS_HARD_STOP_TICKS_SS_8401_DEBUG 2000 /* 30s before hard-stop, for a certain GLES2 test at 128x128 (bound by combined vertex+tiler job) - for issue 8401 */
+#define KBASE_VE_JS_HARD_STOP_TICKS_NSS_DEBUG 100000 /* 1500s (25mins) before NSS hard-stop */
+#define KBASE_VE_JS_RESET_TICKS_SS_DEBUG 500 /* 45s before resetting GPU, for a certain GLES2 test at 128x128 (bound by combined vertex+tiler job) */
+#define KBASE_VE_JS_RESET_TICKS_SS_8401_DEBUG 3000 /* 7.5s before resetting GPU - for issue 8401 */
+#define KBASE_VE_JS_RESET_TICKS_NSS_DEBUG 100166 /* 1502s before resetting GPU */
+
+#define KBASE_VE_JS_SCHEDULING_TICK_NS 2500000000u /* 2.5s */
+#define KBASE_VE_JS_SOFT_STOP_TICKS 1 /* 2.5s before soft-stop a job */
+#define KBASE_VE_JS_HARD_STOP_TICKS_SS 2 /* 5s before hard-stop */
+#define KBASE_VE_JS_HARD_STOP_TICKS_SS_8401 12 /* 30s before hard-stop, for a certain GLES2 test at 128x128 (bound by combined vertex+tiler job) - for issue 8401 */
+#define KBASE_VE_JS_HARD_STOP_TICKS_NSS 600 /* 1500s before NSS hard-stop */
+#define KBASE_VE_JS_RESET_TICKS_SS 3 /* 7.5s before resetting GPU */
+#define KBASE_VE_JS_RESET_TICKS_SS_8401 18 /* 45s before resetting GPU, for a certain GLES2 test at 128x128 (bound by combined vertex+tiler job) - for issue 8401 */
+#define KBASE_VE_JS_RESET_TICKS_NSS 601 /* 1502s before resetting GPU */
+
+#define KBASE_VE_JS_RESET_TIMEOUT_MS 3000 /* 3s before cancelling stuck jobs */
+#define KBASE_VE_JS_CTX_TIMESLICE_NS 1000000 /* 1ms - an agressive timeslice for testing purposes (causes lots of scheduling out for >4 ctxs) */
+#define KBASE_VE_SECURE_BUT_LOSS_OF_PERFORMANCE (uintptr_t)MALI_FALSE /* By default we prefer performance over security on r0p0-15dev0 and KBASE_CONFIG_ATTR_ earlier */
+#define KBASE_VE_POWER_MANAGEMENT_CALLBACKS (uintptr_t)&pm_callbacks
+#define KBASE_VE_MEMORY_RESOURCE_ZBT (uintptr_t)<_zbt
+#define KBASE_VE_MEMORY_RESOURCE_DDR (uintptr_t)<_ddr
+#define KBASE_VE_CPU_SPEED_FUNC (uintptr_t)&kbase_get_vexpress_cpu_clock_speed
+
+/* Set this to 1 to enable dedicated memory banks */
+#define T6F1_ZBT_DDR_ENABLED 0
+#define HARD_RESET_AT_POWER_OFF 0
+
+static kbase_io_resources io_resources =
+{
+ .job_irq_number = 68,
+ .mmu_irq_number = 69,
+ .gpu_irq_number = 70,
+ .io_memory_region =
+ {
+ .start = 0xFC010000,
+ .end = 0xFC010000 + (4096 * 5) - 1
+ }
+};
+
+#if T6F1_ZBT_DDR_ENABLED
+
+static kbase_attribute lt_zbt_attrs[] =
+{
+ {
+ KBASE_MEM_ATTR_PERF_CPU,
+ KBASE_MEM_PERF_SLOW
+ },
+ {
+ KBASE_MEM_ATTR_END,
+ 0
+ }
+};
+
+static kbase_memory_resource lt_zbt =
+{
+ .base = 0xFD000000,
+ .size = 16 * 1024 * 1024UL /* 16MB */,
+ .attributes = lt_zbt_attrs,
+ .name = "T604 ZBT memory"
+};
+
+
+static kbase_attribute lt_ddr_attrs[] =
+{
+ {
+ KBASE_MEM_ATTR_PERF_CPU,
+ KBASE_MEM_PERF_SLOW
+ },
+ {
+ KBASE_MEM_ATTR_END,
+ 0
+ }
+};
+
+static kbase_memory_resource lt_ddr =
+{
+ .base = 0xE0000000,
+ .size = 256 * 1024 * 1024UL /* 256MB */,
+ .attributes = lt_ddr_attrs,
+ .name = "T604 DDR memory"
+};
+
+#endif /* T6F1_ZBT_DDR_ENABLED */
+
+static int pm_callback_power_on(kbase_device *kbdev)
+{
+ /* Nothing is needed on VExpress, but we may have destroyed GPU state (if the below HARD_RESET code is active) */
+ return 1;
+}
+
+static void pm_callback_power_off(kbase_device *kbdev)
+{
+#if HARD_RESET_AT_POWER_OFF
+ /* Cause a GPU hard reset to test whether we have actually idled the GPU
+ * and that we properly reconfigure the GPU on power up.
+ * Usually this would be dangerous, but if the GPU is working correctly it should
+ * be completely safe as the GPU should not be active at this point.
+ * However this is disabled normally because it will most likely interfere with
+ * bus logging etc.
+ */
+ kbase_os_reg_write(kbdev, GPU_CONTROL_REG(GPU_COMMAND), GPU_COMMAND_HARD_RESET);
+#endif
+}
+
+static kbase_pm_callback_conf pm_callbacks =
+{
+ .power_on_callback = pm_callback_power_on,
+ .power_off_callback = pm_callback_power_off
+};
+
+/* Please keep table config_attributes in sync with config_attributes_hw_issue_8408 */
+static kbase_attribute config_attributes[] =
+{
+ {
+ KBASE_CONFIG_ATTR_MEMORY_PER_PROCESS_LIMIT,
+ KBASE_VE_MEMORY_PER_PROCESS_LIMIT
+ },
+ {
+ KBASE_CONFIG_ATTR_UMP_DEVICE,
+ KBASE_VE_UMP_DEVICE
+ },
+
+ {
+ KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_MAX,
+ KBASE_VE_MEMORY_OS_SHARED_MAX
+ },
+
+ {
+ KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_PERF_GPU,
+ KBASE_VE_MEMORY_OS_SHARED_PERF_GPU
+ },
+
+#if T6F1_ZBT_DDR_ENABLED
+ {
+ KBASE_CONFIG_ATTR_MEMORY_RESOURCE,
+ KBASE_VE_MEMORY_RESOURCE_ZBT
+ },
+
+ {
+ KBASE_CONFIG_ATTR_MEMORY_RESOURCE,
+ KBASE_VE_MEMORY_RESOURCE_DDR
+ },
+#endif /* T6F1_ZBT_DDR_ENABLED */
+
+ {
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MAX,
+ KBASE_VE_GPU_FREQ_KHZ_MAX
+ },
+
+ {
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MIN,
+ KBASE_VE_GPU_FREQ_KHZ_MIN
+ },
+
+#if MALI_DEBUG
+/* Use more aggressive scheduling timeouts in debug builds for testing purposes */
+ {
+ KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS,
+ KBASE_VE_JS_SCHEDULING_TICK_NS_DEBUG
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_SOFT_STOP_TICKS,
+ KBASE_VE_JS_SOFT_STOP_TICKS_DEBUG
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS,
+ KBASE_VE_JS_HARD_STOP_TICKS_SS_DEBUG
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS,
+ KBASE_VE_JS_HARD_STOP_TICKS_NSS_DEBUG
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_RESET_TICKS_SS,
+ KBASE_VE_JS_RESET_TICKS_SS_DEBUG
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_RESET_TICKS_NSS,
+ KBASE_VE_JS_RESET_TICKS_NSS_DEBUG
+ },
+#else /* MALI_DEBUG */
+/* In release builds same as the defaults but scaled for 5MHz FPGA */
+ {
+ KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS,
+ KBASE_VE_JS_SCHEDULING_TICK_NS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_SOFT_STOP_TICKS,
+ KBASE_VE_JS_SOFT_STOP_TICKS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS,
+ KBASE_VE_JS_HARD_STOP_TICKS_SS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS,
+ KBASE_VE_JS_HARD_STOP_TICKS_NSS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_RESET_TICKS_SS,
+ KBASE_VE_JS_RESET_TICKS_SS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_RESET_TICKS_NSS,
+ KBASE_VE_JS_RESET_TICKS_NSS
+ },
+#endif /* MALI_DEBUG */
+ {
+ KBASE_CONFIG_ATTR_JS_RESET_TIMEOUT_MS,
+ KBASE_VE_JS_RESET_TIMEOUT_MS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_CTX_TIMESLICE_NS,
+ KBASE_VE_JS_CTX_TIMESLICE_NS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_POWER_MANAGEMENT_CALLBACKS,
+ KBASE_VE_POWER_MANAGEMENT_CALLBACKS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_CPU_SPEED_FUNC,
+ KBASE_VE_CPU_SPEED_FUNC
+ },
+
+ {
+ KBASE_CONFIG_ATTR_SECURE_BUT_LOSS_OF_PERFORMANCE,
+ KBASE_VE_SECURE_BUT_LOSS_OF_PERFORMANCE
+ },
+
+ {
+ KBASE_CONFIG_ATTR_GPU_IRQ_THROTTLE_TIME_US,
+ 20
+ },
+
+ {
+ KBASE_CONFIG_ATTR_END,
+ 0
+ }
+};
+
+/* as config_attributes array above except with different settings for
+ * JS_HARD_STOP_TICKS_SS, JS_RESET_TICKS_SS that
+ * are needed for BASE_HW_ISSUE_8408.
+ */
+kbase_attribute config_attributes_hw_issue_8408[] =
+{
+ {
+ KBASE_CONFIG_ATTR_MEMORY_PER_PROCESS_LIMIT,
+ KBASE_VE_MEMORY_PER_PROCESS_LIMIT
+ },
+ {
+ KBASE_CONFIG_ATTR_UMP_DEVICE,
+ KBASE_VE_UMP_DEVICE
+ },
+
+ {
+ KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_MAX,
+ KBASE_VE_MEMORY_OS_SHARED_MAX
+ },
+
+ {
+ KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_PERF_GPU,
+ KBASE_VE_MEMORY_OS_SHARED_PERF_GPU
+ },
+
+#if T6F1_ZBT_DDR_ENABLED
+ {
+ KBASE_CONFIG_ATTR_MEMORY_RESOURCE,
+ KBASE_VE_MEMORY_RESOURCE_ZBT
+ },
+
+ {
+ KBASE_CONFIG_ATTR_MEMORY_RESOURCE,
+ KBASE_VE_MEMORY_RESOURCE_DDR
+ },
+#endif /* T6F1_ZBT_DDR_ENABLED */
+
+ {
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MAX,
+ KBASE_VE_GPU_FREQ_KHZ_MAX
+ },
+
+ {
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MIN,
+ KBASE_VE_GPU_FREQ_KHZ_MIN
+ },
+
+#if MALI_DEBUG
+/* Use more aggressive scheduling timeouts in debug builds for testing purposes */
+ {
+ KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS,
+ KBASE_VE_JS_SCHEDULING_TICK_NS_DEBUG
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_SOFT_STOP_TICKS,
+ KBASE_VE_JS_SOFT_STOP_TICKS_DEBUG
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS,
+ KBASE_VE_JS_HARD_STOP_TICKS_SS_8401_DEBUG
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS,
+ KBASE_VE_JS_HARD_STOP_TICKS_NSS_DEBUG
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_RESET_TICKS_SS,
+ KBASE_VE_JS_RESET_TICKS_SS_8401_DEBUG
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_RESET_TICKS_NSS,
+ KBASE_VE_JS_RESET_TICKS_NSS_DEBUG
+ },
+#else /* MALI_DEBUG */
+/* In release builds same as the defaults but scaled for 5MHz FPGA */
+ {
+ KBASE_CONFIG_ATTR_JS_SCHEDULING_TICK_NS,
+ KBASE_VE_JS_SCHEDULING_TICK_NS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_SOFT_STOP_TICKS,
+ KBASE_VE_JS_SOFT_STOP_TICKS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS,
+ KBASE_VE_JS_HARD_STOP_TICKS_SS_8401
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS,
+ KBASE_VE_JS_HARD_STOP_TICKS_NSS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_RESET_TICKS_SS,
+ KBASE_VE_JS_RESET_TICKS_SS_8401
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_RESET_TICKS_NSS,
+ KBASE_VE_JS_RESET_TICKS_NSS
+ },
+#endif /* MALI_DEBUG */
+ {
+ KBASE_CONFIG_ATTR_JS_RESET_TIMEOUT_MS,
+ KBASE_VE_JS_RESET_TIMEOUT_MS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_JS_CTX_TIMESLICE_NS,
+ KBASE_VE_JS_CTX_TIMESLICE_NS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_POWER_MANAGEMENT_CALLBACKS,
+ KBASE_VE_POWER_MANAGEMENT_CALLBACKS
+ },
+
+ {
+ KBASE_CONFIG_ATTR_CPU_SPEED_FUNC,
+ KBASE_VE_CPU_SPEED_FUNC
+ },
+
+ {
+ KBASE_CONFIG_ATTR_SECURE_BUT_LOSS_OF_PERFORMANCE,
+ KBASE_VE_SECURE_BUT_LOSS_OF_PERFORMANCE
+ },
+
+ {
+ KBASE_CONFIG_ATTR_END,
+ 0
+ }
+};
+
+kbase_platform_config platform_config =
+{
+ .attributes = config_attributes,
+ .io_resources = &io_resources,
+ .midgard_type = KBASE_MALI_T6F1
+};
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#include <linux/io.h>
+#include <kbase/src/common/mali_kbase.h>
+#include "mali_kbase_cpu_vexpress.h"
+
+#define HZ_IN_MHZ (1000000)
+
+#define CORETILE_EXPRESS_A9X4_SCC_START (0x100E2000)
+#define MOTHERBOARD_SYS_CFG_START (0x10000000)
+#define SYS_CFGDATA_OFFSET (0x000000A0)
+#define SYS_CFGCTRL_OFFSET (0x000000A4)
+#define SYS_CFGSTAT_OFFSET (0x000000A8)
+
+#define SYS_CFGCTRL_START_BIT_VALUE (1 << 31)
+#define READ_REG_BIT_VALUE (0 << 30)
+#define DCC_DEFAULT_BIT_VALUE (0 << 26)
+#define SYS_CFG_OSC_FUNC_BIT_VALUE (1 << 20)
+#define SITE_DEFAULT_BIT_VALUE (1 << 16)
+#define BOARD_STACK_POS_DEFAULT_BIT_VALUE (0 << 12)
+#define DEVICE_DEFAULT_BIT_VALUE (2 << 0)
+#define SYS_CFG_COMPLETE_BIT_VALUE (1 << 0)
+#define SYS_CFG_ERROR_BIT_VALUE (1 << 1)
+
+#define FEED_REG_BIT_MASK (0x0F)
+#define FCLK_PA_DIVIDE_BIT_SHIFT (0x03)
+#define FCLK_PB_DIVIDE_BIT_SHIFT (0x07)
+#define FCLK_PC_DIVIDE_BIT_SHIFT (0x0B)
+#define AXICLK_PA_DIVIDE_BIT_SHIFT (0x0F)
+#define AXICLK_PB_DIVIDE_BIT_SHIFT (0x13)
+
+#define IS_SINGLE_BIT_SET(val,pos) (val&(1<<pos))
+
+/**
+ * kbase_get_vendor_specific_cpu_clock_speed
+ * @brief Retrieves the CPU clock speed.
+ * The implementation is platform specific.
+ * @param[in/out] u32* cpu_clock - the value of CPU clock speed in MHz
+ * @return 0 on success, 1 otherwise
+*/
+int kbase_get_vexpress_cpu_clock_speed(u32* cpu_clock)
+{
+ int result = 0;
+ u32 reg_val = 0;
+ u32 osc2_value = 0;
+ u32 pa_divide = 0;
+ u32 pb_divide = 0;
+ u32 pc_divide = 0;
+ void* volatile pSysCfgReg = 0;
+ void* volatile pSCCReg = 0;
+
+ /* Init the value case something goes wrong */
+ *cpu_clock = 0;
+
+ /* Map CPU register into virtual memory */
+ pSysCfgReg = ioremap(MOTHERBOARD_SYS_CFG_START, 0x1000);
+ if (pSysCfgReg == NULL)
+ {
+ result = 1;
+
+ goto pSysCfgReg_map_failed;
+ }
+
+ pSCCReg = ioremap(CORETILE_EXPRESS_A9X4_SCC_START, 0x1000);
+ if (pSCCReg == NULL)
+ {
+ result = 1;
+
+ goto pSCCReg_map_failed;
+ }
+
+ /*Read SYS regs - OSC2*/
+ reg_val = readl(pSysCfgReg + SYS_CFGCTRL_OFFSET);
+
+ /*Verify if there is no other undergoing request*/
+ if(!(reg_val&SYS_CFGCTRL_START_BIT_VALUE ))
+ {
+ /*Reset the CGFGSTAT reg*/
+ writel(0,(pSysCfgReg + SYS_CFGSTAT_OFFSET));
+
+ writel( SYS_CFGCTRL_START_BIT_VALUE | READ_REG_BIT_VALUE | DCC_DEFAULT_BIT_VALUE |
+ SYS_CFG_OSC_FUNC_BIT_VALUE | SITE_DEFAULT_BIT_VALUE |
+ BOARD_STACK_POS_DEFAULT_BIT_VALUE | DEVICE_DEFAULT_BIT_VALUE,
+ (pSysCfgReg + SYS_CFGCTRL_OFFSET));
+ /* Wait for the transaction to complete */
+ while( !(readl(pSysCfgReg + SYS_CFGSTAT_OFFSET)&SYS_CFG_COMPLETE_BIT_VALUE));
+ /* Read SYS_CFGSTAT Register to get the status of submitted transaction*/
+ reg_val = readl(pSysCfgReg + SYS_CFGSTAT_OFFSET);
+
+ /*------------------------------------------------------------------------------------------*/
+ /* Check for possible errors*/
+ if(reg_val & SYS_CFG_ERROR_BIT_VALUE)
+ {
+ /* Error while setting register*/
+ result = 1;
+ }
+ else
+ {
+ osc2_value = readl(pSysCfgReg + SYS_CFGDATA_OFFSET );
+ /* Read the SCC CFGRW0 register*/
+ reg_val = readl(pSCCReg);
+
+ /*
+ Select the appropriate feed:
+ CFGRW0[0] - CLKOB
+ CFGRW0[1] - CLKOC
+ CFGRW0[2] - FACLK (CLK)B FROM AXICLK PLL)
+ */
+ /* Calculate the FCLK*/
+ if(IS_SINGLE_BIT_SET(reg_val,0)) /*CFGRW0[0] - CLKOB*/
+ {
+ /* CFGRW0[6:3]*/
+ pa_divide =((reg_val&(FEED_REG_BIT_MASK<<FCLK_PA_DIVIDE_BIT_SHIFT))>>FCLK_PA_DIVIDE_BIT_SHIFT);
+ /* CFGRW0[10:7]*/
+ pb_divide =((reg_val&(FEED_REG_BIT_MASK<<FCLK_PB_DIVIDE_BIT_SHIFT))>>FCLK_PB_DIVIDE_BIT_SHIFT);
+ *cpu_clock = osc2_value * (pa_divide + 1) / (pb_divide +1);
+ }
+ else
+ {
+ if(IS_SINGLE_BIT_SET(reg_val,1))/*CFGRW0[1] - CLKOC*/
+ {
+ /* CFGRW0[6:3]*/
+ pa_divide = ((reg_val&(FEED_REG_BIT_MASK<<FCLK_PA_DIVIDE_BIT_SHIFT))>>FCLK_PA_DIVIDE_BIT_SHIFT);
+ /* CFGRW0[14:11]*/
+ pc_divide = ((reg_val&(FEED_REG_BIT_MASK<<FCLK_PC_DIVIDE_BIT_SHIFT)) >> FCLK_PC_DIVIDE_BIT_SHIFT);
+ *cpu_clock = osc2_value * (pa_divide + 1) / (pc_divide + 1);
+ }
+ else
+ if(IS_SINGLE_BIT_SET(reg_val,2))/*CFGRW0[2] - FACLK*/
+ {
+ /* CFGRW0[18:15]*/
+ pa_divide = ((reg_val&(FEED_REG_BIT_MASK<<AXICLK_PA_DIVIDE_BIT_SHIFT)) >>AXICLK_PA_DIVIDE_BIT_SHIFT);
+ /* CFGRW0[22:19]*/
+ pb_divide = ((reg_val&(FEED_REG_BIT_MASK<<AXICLK_PB_DIVIDE_BIT_SHIFT))>>AXICLK_PB_DIVIDE_BIT_SHIFT);
+ *cpu_clock = osc2_value * (pa_divide + 1) / (pb_divide +1);
+ }
+ else
+ {
+ result = 1;
+ }
+ }
+ }
+ }
+ else
+ {
+ result = 1;
+ }
+
+ /* Convert result expressed in Hz to Mhz units.*/
+ *cpu_clock /= HZ_IN_MHZ;
+
+ /* Unmap memory*/
+ iounmap(pSCCReg);
+
+pSCCReg_map_failed:
+ iounmap(pSysCfgReg);
+
+pSysCfgReg_map_failed:
+ return result;
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _KBASE_CPU_VEXPRESS_H_
+#define _KBASE_CPU_VEXPRESS_H_
+
+/**
+ * Versatile Express implementation of @ref kbase_cpuprops_clock_speed_function.
+ */
+int kbase_get_vexpress_cpu_clock_speed(u32* cpu_clock);
+
+#endif /* _KBASE_CPU_VEXPRESS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+#include <kbase/src/linux/mali_kbase_config_linux.h>
+#include <osk/mali_osk.h>
+
+#if !MALI_LICENSE_IS_GPL || MALI_FAKE_PLATFORM_DEVICE
+
+void kbasep_config_parse_io_resources(const kbase_io_resources *io_resources, struct resource *linux_resources)
+{
+ OSK_ASSERT(io_resources != NULL);
+ OSK_ASSERT(linux_resources != NULL);
+
+ OSK_MEMSET(linux_resources, 0, PLATFORM_CONFIG_RESOURCE_COUNT * sizeof(struct resource));
+
+ linux_resources[0].start = io_resources->io_memory_region.start;
+ linux_resources[0].end = io_resources->io_memory_region.end;
+ linux_resources[0].flags = IORESOURCE_MEM;
+
+ linux_resources[1].start = linux_resources[1].end = io_resources->job_irq_number;
+ linux_resources[1].flags = IORESOURCE_IRQ | IORESOURCE_IRQ_HIGHLEVEL;
+
+ linux_resources[2].start = linux_resources[2].end = io_resources->mmu_irq_number;
+ linux_resources[2].flags = IORESOURCE_IRQ | IORESOURCE_IRQ_HIGHLEVEL;
+
+ linux_resources[3].start = linux_resources[3].end = io_resources->gpu_irq_number;
+ linux_resources[3].flags = IORESOURCE_IRQ | IORESOURCE_IRQ_HIGHLEVEL;
+}
+
+#endif /* !MALI_LICENSE_IS_GPL || MALI_FAKE_PLATFORM_DEVICE */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _KBASE_CONFIG_LINUX_H_
+#define _KBASE_CONFIG_LINUX_H_
+
+#include <kbase/mali_kbase_config.h>
+#include <linux/ioport.h>
+
+#define PLATFORM_CONFIG_RESOURCE_COUNT 4
+#define PLATFORM_CONFIG_IRQ_RES_COUNT 3
+
+#if !MALI_LICENSE_IS_GPL || MALI_FAKE_PLATFORM_DEVICE
+/**
+ * @brief Convert data in kbase_io_resources struct to Linux-specific resources
+ *
+ * Function converts data in kbase_io_resources struct to an array of Linux resource structures. Note that function
+ * assumes that size of linux_resource array is at least PLATFORM_CONFIG_RESOURCE_COUNT.
+ * Resources are put in fixed order: I/O memory region, job IRQ, MMU IRQ, GPU IRQ.
+ *
+ * @param[in] io_resource Input IO resource data
+ * @param[out] linux_resources Pointer to output array of Linux resource structures
+ */
+void kbasep_config_parse_io_resources(const kbase_io_resources *io_resource, struct resource *linux_resources);
+#endif /* !MALI_LICENSE_IS_GPL || MALI_FAKE_PLATFORM_DEVICE */
+
+
+#endif /* _KBASE_CONFIG_LINUX_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_core_linux.c
+ * Base kernel driver init.
+ */
+
+#include <osk/mali_osk.h>
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/common/mali_kbase_uku.h>
+#include <kbase/src/common/mali_midg_regmap.h>
+#include <kbase/src/linux/mali_kbase_mem_linux.h>
+#include <kbase/src/linux/mali_kbase_config_linux.h>
+#include <uk/mali_ukk.h>
+#if MALI_NO_MALI
+#include "mali_kbase_model_linux.h"
+#endif
+
+#ifdef CONFIG_KDS
+#include <kds/include/linux/kds.h>
+#include <linux/anon_inodes.h>
+#include <linux/syscalls.h>
+#endif
+
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/poll.h>
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#if MALI_LICENSE_IS_GPL
+#include <linux/platform_device.h>
+#include <linux/miscdevice.h>
+#endif
+#include <linux/list.h>
+#include <linux/semaphore.h>
+#include <linux/fs.h>
+#include <linux/uaccess.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/compat.h> /* is_compat_task */
+#include <kbase/src/common/mali_kbase_8401_workaround.h>
+#include <kbase/src/common/mali_kbase_hw.h>
+
+#if MALI_LICENSE_IS_GPL && MALI_CUSTOMER_RELEASE == 0 && MALI_COVERAGE == 0
+#include <linux/pci.h>
+#define MALI_PCI_DEVICE
+#endif
+
+#define JOB_IRQ_TAG 0
+#define MMU_IRQ_TAG 1
+#define GPU_IRQ_TAG 2
+
+struct kbase_irq_table
+{
+ u32 tag;
+ irq_handler_t handler;
+};
+#if MALI_UNIT_TEST
+kbase_exported_test_data shared_kernel_test_data;
+EXPORT_SYMBOL(shared_kernel_test_data);
+#endif /* MALI_UNIT_TEST */
+
+static const char kbase_drv_name[] = KBASE_DRV_NAME;
+
+static int kbase_dev_nr;
+
+#if MALI_LICENSE_IS_GPL
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,36)
+static DEFINE_SEMAPHORE(kbase_dev_list_lock);
+#else
+static DECLARE_MUTEX(kbase_dev_list_lock);
+#endif
+static LIST_HEAD(kbase_dev_list);
+
+KBASE_EXPORT_TEST_API(kbase_dev_list_lock)
+KBASE_EXPORT_TEST_API(kbase_dev_list)
+#endif
+
+#if MALI_LICENSE_IS_GPL == 0
+#include <linux/cdev.h> /* character device definitions */
+
+/* By default the module uses any available major, but it's possible to set it at load time to a specific number */
+int mali_major = 0;
+module_param(mali_major, int, S_IRUGO); /* r--r--r-- */
+MODULE_PARM_DESC(mali_major, "Device major number");
+
+struct mali_linux_device
+{
+ struct cdev cdev;
+};
+
+/* The global variable containing the global device data */
+static struct mali_linux_device mali_linux_device;
+
+static char mali_dev_name[] = KBASE_DRV_NAME; /* should be const, but the functions we call requires non-cost */
+
+#undef dev_err
+#undef dev_info
+#undef dev_dbg
+#define dev_err(dev,msg,...) do { printk(KERN_ERR KBASE_DRV_NAME " error: "); printk(msg, ## __VA_ARGS__); } while(0)
+#define dev_info(dev,msg,...) do { printk(KERN_INFO KBASE_DRV_NAME " info: "); printk(msg, ## __VA_ARGS__); } while(0)
+#define dev_dbg(dev,msg,...) do { printk(KERN_DEBUG KBASE_DRV_NAME " debug: "); printk(msg, ## __VA_ARGS__); } while(0)
+#define dev_name(dev) "MALI"
+
+/* STATIC */ struct kbase_device *g_kbdev;
+KBASE_EXPORT_TEST_API(g_kbdev);
+
+#endif
+
+
+#if MALI_LICENSE_IS_GPL
+#define KERNEL_SIDE_DDK_VERSION_STRING "K:" MALI_RELEASE_NAME "(GPL)"
+#else
+#define KERNEL_SIDE_DDK_VERSION_STRING "K:" MALI_RELEASE_NAME
+#endif /* MALI_LICENSE_IS_GPL */
+
+static INLINE void __compile_time_asserts( void )
+{
+ CSTD_COMPILE_TIME_ASSERT( sizeof(KERNEL_SIDE_DDK_VERSION_STRING) <= KBASE_GET_VERSION_BUFFER_SIZE);
+}
+
+#ifdef CONFIG_KDS
+
+typedef struct kbasep_kds_resource_set_file_data
+{
+ struct kds_resource_set * lock;
+}kbasep_kds_resource_set_file_data;
+
+static int kds_resource_release(struct inode *inode, struct file *file);
+
+static const struct file_operations kds_resource_fops =
+{
+ .release = kds_resource_release
+};
+
+typedef struct kbase_kds_resource_list_data
+{
+ struct kds_resource ** kds_resources;
+ unsigned long * kds_access_bitmap;
+ int num_elems;
+}kbase_kds_resource_list_data;
+
+
+static int kds_resource_release(struct inode *inode, struct file *file)
+{
+ struct kbasep_kds_resource_set_file_data *data;
+
+ data = (struct kbasep_kds_resource_set_file_data *)file->private_data;
+ if ( NULL != data )
+ {
+ if ( NULL != data->lock )
+ {
+ kds_resource_set_release( &data->lock );
+ }
+ osk_free( data );
+ }
+ return 0;
+}
+
+mali_error kbasep_kds_allocate_resource_list_data( kbase_context * kctx,
+ base_external_resource *ext_res,
+ int num_elems,
+ kbase_kds_resource_list_data * resources_list )
+{
+ base_external_resource *res = ext_res;
+ int res_id;
+
+ /* assume we have to wait for all */
+ resources_list->kds_resources = osk_malloc(sizeof(struct kds_resource *) * num_elems);
+ if ( NULL == resources_list->kds_resources )
+ {
+ return MALI_ERROR_OUT_OF_MEMORY;
+ }
+
+ resources_list->kds_access_bitmap = osk_calloc(sizeof(unsigned long) * ((num_elems + OSK_BITS_PER_LONG - 1) / OSK_BITS_PER_LONG));
+ if (NULL == resources_list->kds_access_bitmap)
+ {
+ osk_free(resources_list->kds_access_bitmap);
+ return MALI_ERROR_OUT_OF_MEMORY;
+ }
+
+ for (res_id = 0; res_id < num_elems; res_id++, res++ )
+ {
+ int exclusive;
+ kbase_va_region * reg;
+ struct kds_resource * kds_res = NULL;
+
+ exclusive = res->ext_resource & BASE_EXT_RES_ACCESS_EXCLUSIVE;
+ reg = kbase_region_lookup(kctx, res->ext_resource & ~BASE_EXT_RES_ACCESS_EXCLUSIVE);
+
+ /* did we find a matching region object? */
+ if (NULL == reg)
+ {
+ break;
+ }
+
+ switch (reg->imported_type)
+ {
+#if MALI_USE_UMP == 1
+ case BASE_TMEM_IMPORT_TYPE_UMP:
+ kds_res = ump_dd_kds_resource_get(reg->imported_metadata.ump_handle);
+ break;
+#endif /*MALI_USE_UMP == 1*/
+ default:
+ break;
+ }
+
+ /* no kds resource for the region ? */
+ if (!kds_res)
+ {
+ break;
+ }
+
+ resources_list->kds_resources[res_id] = kds_res;
+
+ if (exclusive)
+ {
+ osk_bitarray_set_bit(res_id, resources_list->kds_access_bitmap);
+ }
+ }
+
+ /* did the loop run to completion? */
+ if (res_id == num_elems)
+ {
+ return MALI_ERROR_NONE;
+ }
+
+ /* Clean up as the resource list is not valid. */
+ osk_free( resources_list->kds_resources );
+ osk_free( resources_list->kds_access_bitmap );
+
+ return MALI_ERROR_FUNCTION_FAILED;
+}
+
+mali_bool kbasep_validate_kbase_pointer( kbase_pointer * p )
+{
+#ifdef CONFIG_COMPAT
+ if (is_compat_task())
+ {
+ if ( p->compat_value == 0 )
+ {
+ return MALI_FALSE;
+ }
+ }
+ else
+ {
+#endif /* CONFIG_COMPAT */
+ if ( NULL == p->value )
+ {
+ return MALI_FALSE;
+ }
+#ifdef CONFIG_COMPAT
+ }
+#endif /* CONFIG_COMPAT */
+ return MALI_TRUE;
+}
+
+mali_error kbase_external_buffer_lock(kbase_context * kctx, ukk_call_context *ukk_ctx, kbase_uk_ext_buff_kds_data *args, u32 args_size)
+{
+ base_external_resource *ext_res_copy;
+ size_t ext_resource_size;
+ mali_error return_error = MALI_ERROR_FUNCTION_FAILED;
+ int fd;
+
+ if (args_size != sizeof(kbase_uk_ext_buff_kds_data))
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ /* Check user space has provided valid data */
+ if ( !kbasep_validate_kbase_pointer(&args->external_resource) ||
+ !kbasep_validate_kbase_pointer(&args->file_descriptor) ||
+ (0 == args->num_res))
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ ext_resource_size = sizeof( base_external_resource ) * args->num_res;
+ ext_res_copy = (base_external_resource *)osk_malloc( ext_resource_size );
+
+ if ( NULL != ext_res_copy )
+ {
+ base_external_resource * __user ext_res_user;
+ int * __user file_descriptor_user;
+#ifdef CONFIG_COMPAT
+ if (is_compat_task())
+ {
+ ext_res_user = args->external_resource.compat_value;
+ file_descriptor_user = args->file_descriptor.compat_value;
+ }
+ else
+ {
+#endif /* CONFIG_COMPAT */
+ ext_res_user = args->external_resource.value;
+ file_descriptor_user = args->file_descriptor.value;
+#ifdef CONFIG_COMPAT
+ }
+#endif /* CONFIG_COMPAT */
+
+ /* Copy the external resources to lock from user space */
+ if ( MALI_ERROR_NONE == ukk_copy_from_user( ext_resource_size, ext_res_copy, ext_res_user ) )
+ {
+ /* Allocate data to be stored in the file */
+ kbasep_kds_resource_set_file_data * fdata = osk_malloc( sizeof( kbasep_kds_resource_set_file_data));
+
+ if ( NULL != fdata )
+ {
+ kbase_kds_resource_list_data resource_list_data;
+ /* Parse given elements and create resource and access lists */
+ return_error = kbasep_kds_allocate_resource_list_data( kctx, ext_res_copy, args->num_res, &resource_list_data );
+ if ( MALI_ERROR_NONE == return_error )
+ {
+ fdata->lock = NULL;
+
+ fd = anon_inode_getfd( "kds_ext", &kds_resource_fops, fdata, 0 );
+
+ return_error = ukk_copy_to_user( sizeof( fd ), file_descriptor_user, &fd );
+
+ /* If the file descriptor was valid and we successfully copied it to user space, then we
+ * can try and lock the requested kds resources.
+ */
+ if ( ( fd >= 0 ) && ( MALI_ERROR_NONE == return_error ) )
+ {
+ struct kds_resource_set * lock;
+
+ lock = kds_waitall(args->num_res,
+ resource_list_data.kds_access_bitmap,
+ resource_list_data.kds_resources, KDS_WAIT_BLOCKING );
+
+ if (IS_ERR_OR_NULL(lock))
+ {
+ return_error = MALI_ERROR_FUNCTION_FAILED;
+ }
+ else
+ {
+ return_error = MALI_ERROR_NONE;
+ fdata->lock = lock;
+ }
+ }
+ else
+ {
+ return_error = MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ osk_free( resource_list_data.kds_resources );
+ osk_free( resource_list_data.kds_access_bitmap );
+ }
+
+ if ( MALI_ERROR_NONE != return_error )
+ {
+ /* If the file was opened successfully then close it which will clean up
+ * the file data, otherwise we clean up the file data ourself. */
+ if ( fd >= 0 )
+ {
+ sys_close(fd);
+ }
+ else
+ {
+ osk_free( fdata );
+ }
+ }
+ }
+ else
+ {
+ return_error = MALI_ERROR_OUT_OF_MEMORY;
+ }
+ }
+ osk_free( ext_res_copy );
+ }
+ return return_error;
+}
+#endif
+
+
+static mali_error kbase_dispatch(ukk_call_context * const ukk_ctx, void * const args, u32 args_size)
+{
+ struct kbase_context *kctx;
+ struct kbase_device *kbdev;
+ uk_header *ukh = args;
+ u32 id;
+
+ OSKP_ASSERT( ukh != NULL );
+
+ kctx = CONTAINER_OF(ukk_session_get(ukk_ctx), kbase_context, ukk_session);
+ kbdev = kctx->kbdev;
+ id = ukh->id;
+ ukh->ret = MALI_ERROR_NONE; /* Be optimistic */
+
+ switch(id)
+ {
+ case KBASE_FUNC_TMEM_ALLOC:
+ {
+ kbase_uk_tmem_alloc *tmem = args;
+ struct kbase_va_region *reg;
+
+ if (sizeof(*tmem) != args_size)
+ {
+ goto bad_size;
+ }
+
+ reg = kbase_tmem_alloc(kctx, tmem->vsize, tmem->psize,
+ tmem->extent, tmem->flags, tmem->is_growable);
+ if (reg)
+ {
+ tmem->gpu_addr = reg->start_pfn << OSK_PAGE_SHIFT;
+ }
+ else
+ {
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ break;
+ }
+
+ case KBASE_FUNC_TMEM_IMPORT:
+ {
+ kbase_uk_tmem_import * tmem_import = args;
+ struct kbase_va_region *reg;
+ int * __user phandle;
+ int handle;
+
+ if (sizeof(*tmem_import) != args_size)
+ {
+ goto bad_size;
+ }
+#ifdef CONFIG_COMPAT
+ if (is_compat_task())
+ {
+ phandle = tmem_import->phandle.compat_value;
+ }
+ else
+ {
+#endif /* CONFIG_COMPAT */
+ phandle = tmem_import->phandle.value;
+#ifdef CONFIG_COMPAT
+ }
+#endif /* CONFIG_COMPAT */
+
+ /* code should be in kbase_tmem_import and its helpers, but uk dropped its get_user abstraction */
+ switch (tmem_import->type)
+ {
+#if MALI_USE_UMP == 1
+ case BASE_TMEM_IMPORT_TYPE_UMP:
+ get_user(handle, phandle);
+ break;
+#endif /* MALI_USE_UMP == 1 */
+ case BASE_TMEM_IMPORT_TYPE_UMM:
+ get_user(handle, phandle);
+ break;
+ default:
+ goto bad_type;
+ break;
+ }
+
+ reg = kbase_tmem_import(kctx, tmem_import->type, handle, &tmem_import->pages);
+
+ if (reg)
+ {
+ tmem_import->gpu_addr = reg->start_pfn << OSK_PAGE_SHIFT;
+ }
+ else
+ {
+bad_type:
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ break;
+ }
+ case KBASE_FUNC_PMEM_ALLOC:
+ {
+ kbase_uk_pmem_alloc *pmem = args;
+ struct kbase_va_region *reg;
+
+ if (sizeof(*pmem) != args_size)
+ {
+ goto bad_size;
+ }
+
+ reg = kbase_pmem_alloc(kctx, pmem->vsize, pmem->flags,
+ &pmem->cookie);
+ if (!reg)
+ {
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ break;
+ }
+
+ case KBASE_FUNC_MEM_FREE:
+ {
+ kbase_uk_mem_free *mem = args;
+
+ if (sizeof(*mem) != args_size)
+ {
+ goto bad_size;
+ }
+
+ if ((mem->gpu_addr & BASE_MEM_TAGS_MASK)&&(mem->gpu_addr >= OSK_PAGE_SIZE))
+ {
+ OSK_PRINT_WARN(OSK_BASE_MEM, "kbase_dispatch case KBASE_FUNC_MEM_FREE: mem->gpu_addr: passed parameter is invalid");
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ break;
+ }
+
+ if (kbase_mem_free(kctx, mem->gpu_addr))
+ {
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ break;
+ }
+
+ case KBASE_FUNC_JOB_SUBMIT:
+ {
+ kbase_uk_job_submit * job = args;
+
+ if (sizeof(*job) != args_size)
+ {
+ goto bad_size;
+ }
+
+ if (MALI_ERROR_NONE != kbase_jd_submit(kctx, job))
+ {
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ break;
+ }
+
+ case KBASE_FUNC_SYNC:
+ {
+ kbase_uk_sync_now *sn = args;
+
+ if (sizeof(*sn) != args_size)
+ {
+ goto bad_size;
+ }
+
+ if (sn->sset.basep_sset.mem_handle & BASE_MEM_TAGS_MASK)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MEM, "kbase_dispatch case KBASE_FUNC_SYNC: sn->sset.basep_sset.mem_handle: passed parameter is invalid");
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ break;
+ }
+
+ if (MALI_ERROR_NONE != kbase_sync_now(kctx, &sn->sset))
+ {
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ break;
+ }
+
+ case KBASE_FUNC_POST_TERM:
+ {
+ kbase_event_close(kctx);
+ break;
+ }
+
+ case KBASE_FUNC_HWCNT_SETUP:
+ {
+ kbase_uk_hwcnt_setup * setup = args;
+
+ if (sizeof(*setup) != args_size)
+ {
+ goto bad_size;
+ }
+ if (MALI_ERROR_NONE != kbase_instr_hwcnt_setup(kctx, setup))
+ {
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ break;
+ }
+
+ case KBASE_FUNC_HWCNT_DUMP:
+ {
+ /* args ignored */
+ if (MALI_ERROR_NONE != kbase_instr_hwcnt_dump(kctx))
+ {
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ break;
+ }
+
+ case KBASE_FUNC_HWCNT_CLEAR:
+ {
+ /* args ignored */
+ if (MALI_ERROR_NONE != kbase_instr_hwcnt_clear(kctx))
+ {
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ break;
+ }
+
+ case KBASE_FUNC_CPU_PROPS_REG_DUMP:
+ {
+ kbase_uk_cpuprops * setup = args;
+
+ if (sizeof(*setup) != args_size)
+ {
+ goto bad_size;
+ }
+
+ if (MALI_ERROR_NONE != kbase_cpuprops_uk_get_props(kctx,setup))
+ {
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ break;
+ }
+
+ case KBASE_FUNC_GPU_PROPS_REG_DUMP:
+ {
+ kbase_uk_gpuprops * setup = args;
+
+ if (sizeof(*setup) != args_size)
+ {
+ goto bad_size;
+ }
+
+ if (MALI_ERROR_NONE != kbase_gpuprops_uk_get_props(kctx, setup))
+ {
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ break;
+ }
+
+ case KBASE_FUNC_TMEM_RESIZE:
+ {
+ kbase_uk_tmem_resize *resize = args;
+ if (sizeof(*resize) != args_size)
+ {
+ goto bad_size;
+ }
+ if (resize->gpu_addr & BASE_MEM_TAGS_MASK)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MEM, "kbase_dispatch case KBASE_FUNC_TMEM_RESIZE: resize->gpu_addr: passed parameter is invalid");
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ break;
+ }
+
+ ukh->ret = kbase_tmem_resize(kctx, resize->gpu_addr, resize->delta, &resize->size, &resize->result_subcode);
+ break;
+ }
+
+ case KBASE_FUNC_FIND_CPU_MAPPING:
+ {
+ kbase_uk_find_cpu_mapping *find = args;
+ struct kbase_cpu_mapping *map;
+
+ if (sizeof(*find) != args_size)
+ {
+ goto bad_size;
+ }
+ if (find->gpu_addr & BASE_MEM_TAGS_MASK)
+ {
+ OSK_PRINT_WARN(OSK_BASE_MEM, "kbase_dispatch case KBASE_FUNC_FIND_CPU_MAPPING: find->gpu_addr: passed parameter is invalid");
+ goto out_bad;
+ }
+
+ OSKP_ASSERT( find != NULL );
+ if ( find->size > SIZE_MAX || find->cpu_addr > UINTPTR_MAX )
+ {
+ map = NULL;
+ }
+ else
+ {
+ map = kbasep_find_enclosing_cpu_mapping( kctx,
+ find->gpu_addr,
+ (osk_virt_addr)(uintptr_t)find->cpu_addr,
+ (size_t)find->size );
+ }
+
+ if ( NULL != map )
+ {
+ find->uaddr = PTR_TO_U64( map->uaddr );
+ find->nr_pages = map->nr_pages;
+ find->page_off = map->page_off;
+ }
+ else
+ {
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ break;
+ }
+ case KBASE_FUNC_GET_VERSION:
+ {
+ kbase_uk_get_ddk_version *get_version = (kbase_uk_get_ddk_version *)args;
+
+ if (sizeof(*get_version) != args_size)
+ {
+ goto bad_size;
+ }
+
+ /* version buffer size check is made in compile time assert */
+ OSK_MEMCPY(get_version->version_buffer, KERNEL_SIDE_DDK_VERSION_STRING,
+ sizeof(KERNEL_SIDE_DDK_VERSION_STRING));
+ get_version->version_string_size = sizeof(KERNEL_SIDE_DDK_VERSION_STRING);
+ break;
+ }
+#ifdef CONFIG_KDS
+ case KBASE_FUNC_EXT_BUFFER_LOCK:
+ {
+ ukh->ret = kbase_external_buffer_lock( kctx, ukk_ctx,(kbase_uk_ext_buff_kds_data *)args, args_size );
+ break;
+ }
+#endif
+ case KBASE_FUNC_SET_FLAGS:
+ {
+ kbase_uk_set_flags *kbase_set_flags = (kbase_uk_set_flags *)args;
+
+ if (sizeof(*kbase_set_flags) != args_size)
+ {
+ goto bad_size;
+ }
+
+ if (MALI_ERROR_NONE != kbase_context_set_create_flags(kctx, kbase_set_flags->create_flags))
+ {
+ ukh->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ break;
+ }
+
+#if MALI_UNIT_TEST
+ case KBASE_FUNC_SET_TEST_DATA:
+ {
+ kbase_uk_set_test_data *set_data = args;
+
+ shared_kernel_test_data = set_data->test_data;
+ shared_kernel_test_data.kctx = kctx;
+ shared_kernel_test_data.mm = (void*)current->mm;
+ ukh->ret = MALI_ERROR_NONE;
+ break;
+ }
+#endif /* MALI_UNIT_TEST */
+#if MALI_ERROR_INJECT_ON
+ case KBASE_FUNC_INJECT_ERROR:
+ {
+ kbase_error_params params = ((kbase_uk_error_params*)args)->params;
+ /*mutex lock*/
+ osk_spinlock_lock(&kbdev->osdev.reg_op_lock);
+ ukh->ret = job_atom_inject_error(¶ms);
+ osk_spinlock_unlock(&kbdev->osdev.reg_op_lock);
+ /*mutex unlock*/
+
+ break;
+ }
+#endif /*MALI_ERROR_INJECT_ON*/
+#if MALI_NO_MALI
+ case KBASE_FUNC_MODEL_CONTROL:
+ {
+ kbase_model_control_params params = ((kbase_uk_model_control_params*)args)->params;
+ /*mutex lock*/
+ osk_spinlock_lock(&kbdev->osdev.reg_op_lock);
+ ukh->ret = midg_model_control(kbdev->osdev.model, ¶ms);
+ osk_spinlock_unlock(&kbdev->osdev.reg_op_lock);
+ /*mutex unlock*/
+ break;
+ }
+#endif /* MALI_NO_MALI */
+ default:
+ dev_err(kbdev->osdev.dev, "unknown syscall %08x", ukh->id);
+ goto out_bad;
+ }
+
+ return MALI_ERROR_NONE;
+
+bad_size:
+ dev_err(kbdev->osdev.dev, "Wrong syscall size (%d) for %08x\n", args_size, ukh->id);
+out_bad:
+ return MALI_ERROR_FUNCTION_FAILED;
+}
+
+#if MALI_LICENSE_IS_GPL
+static struct kbase_device *to_kbase_device(struct device *dev)
+{
+ return dev_get_drvdata(dev);
+}
+#endif /* MALI_LICENSE_IS_GPL */
+
+/* Find a particular kbase device (as specified by minor number), or find the "first" device if -1 is specified */
+struct kbase_device *kbase_find_device(int minor)
+{
+ struct kbase_device *kbdev = NULL;
+#if MALI_LICENSE_IS_GPL
+ struct list_head *entry;
+
+ down(&kbase_dev_list_lock);
+ list_for_each(entry, &kbase_dev_list)
+ {
+ struct kbase_device *tmp;
+
+ tmp = list_entry(entry, struct kbase_device, osdev.entry);
+ if (tmp->osdev.mdev.minor == minor || minor == -1)
+ {
+ kbdev = tmp;
+ get_device(kbdev->osdev.dev);
+ break;
+ }
+ }
+ up(&kbase_dev_list_lock);
+#else
+ kbdev = g_kbdev;
+#endif
+
+ return kbdev;
+}
+
+EXPORT_SYMBOL(kbase_find_device);
+
+
+
+
+
+void kbase_release_device(struct kbase_device *kbdev)
+{
+#if MALI_LICENSE_IS_GPL
+ put_device(kbdev->osdev.dev);
+#endif
+}
+
+EXPORT_SYMBOL(kbase_release_device);
+
+static int kbase_open(struct inode *inode, struct file *filp)
+{
+ struct kbase_device *kbdev = NULL;
+ struct kbase_context *kctx;
+ int ret = 0;
+
+ /* Enforce that the driver is opened with O_CLOEXEC so that execve() automatically
+ * closes the file descriptor in a child process.
+ */
+ if (0 == (filp->f_flags & O_CLOEXEC))
+ {
+ printk(KERN_ERR KBASE_DRV_NAME " error: O_CLOEXEC flag not set\n");
+ return -EINVAL;
+ }
+
+ kbdev = kbase_find_device(iminor(inode));
+
+ if (!kbdev)
+ return -ENODEV;
+
+ kctx = kbase_create_context(kbdev);
+ if (!kctx)
+ {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ if (MALI_ERROR_NONE != ukk_session_init(&kctx->ukk_session, kbase_dispatch, BASE_UK_VERSION_MAJOR, BASE_UK_VERSION_MINOR))
+ {
+ kbase_destroy_context(kctx);
+ ret = -EFAULT;
+ goto out;
+ }
+
+ init_waitqueue_head(&kctx->osctx.event_queue);
+ filp->private_data = kctx;
+
+ dev_dbg(kbdev->osdev.dev, "created base context\n");
+ return 0;
+
+out:
+ kbase_release_device(kbdev);
+ return ret;
+}
+
+static int kbase_release(struct inode *inode, struct file *filp)
+{
+ struct kbase_context *kctx = filp->private_data;
+ struct kbase_device *kbdev = kctx->kbdev;
+
+ ukk_session_term(&kctx->ukk_session);
+ filp->private_data = NULL;
+ kbase_destroy_context(kctx);
+
+ dev_dbg(kbdev->osdev.dev, "deleted base context\n");
+ kbase_release_device(kbdev);
+ return 0;
+}
+
+static long kbase_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
+{
+ u64 msg[(UKK_CALL_MAX_SIZE+7)>>3]; /* alignment fixup */
+ u32 size = _IOC_SIZE(cmd);
+ ukk_call_context ukk_ctx;
+ struct kbase_context *kctx = filp->private_data;
+
+ if (size > UKK_CALL_MAX_SIZE) return -ENOTTY;
+
+ if (0 != copy_from_user(&msg, (void *)arg, size))
+ {
+ return -EFAULT;
+ }
+
+ ukk_call_prepare(&ukk_ctx, &kctx->ukk_session);
+
+ if (MALI_ERROR_NONE != ukk_dispatch(&ukk_ctx, &msg, size))
+ {
+ return -EFAULT;
+ }
+
+ if (0 != copy_to_user((void *)arg, &msg, size))
+ {
+ pr_err("failed to copy results of UK call back to user space\n");
+ return -EFAULT;
+ }
+ return 0;
+}
+
+static ssize_t kbase_read(struct file *filp, char __user *buf,
+ size_t count, loff_t *f_pos)
+{
+ struct kbase_context *kctx = filp->private_data;
+ base_jd_event uevent;
+
+ if (count < sizeof(uevent))
+ {
+ return -ENOBUFS;
+ }
+
+ while (kbase_event_dequeue(kctx, &uevent))
+ {
+ if (filp->f_flags & O_NONBLOCK)
+ {
+ return -EAGAIN;
+ }
+
+ if (wait_event_interruptible(kctx->osctx.event_queue,
+ kbase_event_pending(kctx)))
+ {
+ return -ERESTARTSYS;
+ }
+ }
+
+ if (copy_to_user(buf, &uevent, sizeof(uevent)))
+ {
+ return -EFAULT;
+ }
+
+ return sizeof(uevent);
+}
+
+static unsigned int kbase_poll(struct file *filp, poll_table *wait)
+{
+ struct kbase_context *kctx = filp->private_data;
+
+ poll_wait(filp, &kctx->osctx.event_queue, wait);
+ if (kbase_event_pending(kctx))
+ {
+ return POLLIN | POLLRDNORM;
+ }
+
+ return 0;
+}
+
+void kbase_event_wakeup(kbase_context *kctx)
+{
+ OSK_ASSERT(kctx);
+
+ wake_up_interruptible(&kctx->osctx.event_queue);
+}
+KBASE_EXPORT_TEST_API(kbase_event_wakeup)
+
+int kbase_check_flags(int flags)
+{
+ /* Enforce that the driver keeps the O_CLOEXEC flag so that execve() always
+ * closes the file descriptor in a child process.
+ */
+ if (0 == (flags & O_CLOEXEC))
+ {
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static const struct file_operations kbase_fops =
+{
+ .owner = THIS_MODULE,
+ .open = kbase_open,
+ .release = kbase_release,
+ .read = kbase_read,
+ .poll = kbase_poll,
+ .unlocked_ioctl = kbase_ioctl,
+ .mmap = kbase_mmap,
+ .check_flags = kbase_check_flags,
+};
+
+#if !MALI_NO_MALI
+void kbase_os_reg_write(kbase_device *kbdev, u16 offset, u32 value)
+{
+ writel(value, kbdev->osdev.reg + offset);
+}
+
+u32 kbase_os_reg_read(kbase_device *kbdev, u16 offset)
+{
+ return readl(kbdev->osdev.reg + offset);
+}
+
+static void *kbase_tag(void *ptr, u32 tag)
+{
+ return (void *)(((uintptr_t) ptr) | tag);
+}
+
+static void *kbase_untag(void *ptr)
+{
+ return (void *)(((uintptr_t) ptr) & ~3);
+}
+
+static irqreturn_t kbase_job_irq_handler(int irq, void *data)
+{
+ struct kbase_device *kbdev = kbase_untag(data);
+ u32 val;
+
+ osk_spinlock_irq_lock(&kbdev->pm.gpu_powered_lock);
+
+ if (!kbdev->pm.gpu_powered)
+ {
+ /* GPU is turned off - IRQ is not for us */
+ osk_spinlock_irq_unlock(&kbdev->pm.gpu_powered_lock);
+ return IRQ_NONE;
+ }
+
+ val = kbase_reg_read(kbdev, JOB_CONTROL_REG(JOB_IRQ_STATUS), NULL);
+
+ osk_spinlock_irq_unlock(&kbdev->pm.gpu_powered_lock);
+
+ if (!val)
+ {
+ return IRQ_NONE;
+ }
+
+ dev_dbg(kbdev->osdev.dev, "%s: irq %d irqstatus 0x%x\n", __func__, irq, val);
+
+ kbase_job_done(kbdev, val);
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t kbase_mmu_irq_handler(int irq, void *data)
+{
+ struct kbase_device *kbdev = kbase_untag(data);
+ u32 val;
+
+ osk_spinlock_irq_lock(&kbdev->pm.gpu_powered_lock);
+
+ if (!kbdev->pm.gpu_powered)
+ {
+ /* GPU is turned off - IRQ is not for us */
+ osk_spinlock_irq_unlock(&kbdev->pm.gpu_powered_lock);
+ return IRQ_NONE;
+ }
+
+ val = kbase_reg_read(kbdev, MMU_REG(MMU_IRQ_STATUS), NULL);
+
+ osk_spinlock_irq_unlock(&kbdev->pm.gpu_powered_lock);
+
+ if (!val)
+ {
+ return IRQ_NONE;
+ }
+
+ dev_dbg(kbdev->osdev.dev, "%s: irq %d irqstatus 0x%x\n", __func__, irq, val);
+
+ kbase_mmu_interrupt(kbdev, val);
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t kbase_gpu_irq_handler(int irq, void *data)
+{
+ struct kbase_device *kbdev = kbase_untag(data);
+ u32 val;
+
+ osk_spinlock_irq_lock(&kbdev->pm.gpu_powered_lock);
+
+ if (!kbdev->pm.gpu_powered)
+ {
+ /* GPU is turned off - IRQ is not for us */
+ osk_spinlock_irq_unlock(&kbdev->pm.gpu_powered_lock);
+ return IRQ_NONE;
+ }
+
+ val = kbase_reg_read(kbdev, GPU_CONTROL_REG(GPU_IRQ_STATUS), NULL);
+
+ osk_spinlock_irq_unlock(&kbdev->pm.gpu_powered_lock);
+
+ if (!val)
+ {
+ return IRQ_NONE;
+ }
+
+ dev_dbg(kbdev->osdev.dev, "%s: irq %d irqstatus 0x%x\n", __func__, irq, val);
+
+ kbase_gpu_interrupt(kbdev, val);
+
+ return IRQ_HANDLED;
+}
+
+static irq_handler_t kbase_handler_table[] = {
+ [JOB_IRQ_TAG] = kbase_job_irq_handler,
+ [MMU_IRQ_TAG] = kbase_mmu_irq_handler,
+ [GPU_IRQ_TAG] = kbase_gpu_irq_handler,
+};
+
+static int kbase_install_interrupts(kbase_device *kbdev)
+{
+ struct kbase_os_device *osdev = &kbdev->osdev;
+ u32 nr = ARRAY_SIZE(kbase_handler_table);
+ int err;
+ u32 i;
+
+ BUG_ON(nr > PLATFORM_CONFIG_IRQ_RES_COUNT); /* Only 3 interrupts! */
+
+ for (i = 0; i < nr; i++)
+ {
+ err = request_irq(osdev->irqs[i].irq,
+ kbase_handler_table[i],
+ osdev->irqs[i].flags | IRQF_SHARED,
+ dev_name(osdev->dev),
+ kbase_tag(kbdev, i));
+ if (err)
+ {
+ dev_err(osdev->dev, "Can't request interrupt %d (index %d)\n", osdev->irqs[i].irq, i);
+ goto release;
+ }
+ }
+
+ return 0;
+
+release:
+ while (i-- > 0)
+ {
+ free_irq(osdev->irqs[i].irq, kbase_tag(kbdev, i));
+ }
+
+ return err;
+}
+
+static void kbase_release_interrupts(kbase_device *kbdev)
+{
+ struct kbase_os_device *osdev = &kbdev->osdev;
+ u32 nr = ARRAY_SIZE(kbase_handler_table);
+ u32 i;
+
+ for (i = 0; i < nr; i++)
+ {
+ if (osdev->irqs[i].irq)
+ {
+ free_irq(osdev->irqs[i].irq, kbase_tag(kbdev, i));
+ }
+ }
+}
+
+void kbase_synchronize_irqs(kbase_device *kbdev)
+{
+ struct kbase_os_device *osdev = &kbdev->osdev;
+ u32 nr = ARRAY_SIZE(kbase_handler_table);
+ u32 i;
+
+ for (i = 0; i < nr; i++)
+ {
+ if (osdev->irqs[i].irq)
+ {
+ synchronize_irq(osdev->irqs[i].irq);
+ }
+ }
+}
+#endif
+
+#if MALI_LICENSE_IS_GPL
+
+/** Show callback for the @c power_policy sysfs file.
+ *
+ * This function is called to get the contents of the @c power_policy sysfs
+ * file. This is a list of the available policies with the currently active one
+ * surrounded by square brackets.
+ *
+ * @param dev The device this sysfs file is for
+ * @param attr The attributes of the sysfs file
+ * @param buf The output buffer for the sysfs file contents
+ *
+ * @return The number of bytes output to @c buf.
+ */
+static ssize_t show_policy(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct kbase_device *kbdev;
+ const struct kbase_pm_policy *current_policy;
+ const struct kbase_pm_policy * const *policy_list;
+ int policy_count;
+ int i;
+ ssize_t ret = 0;
+
+ kbdev = to_kbase_device(dev);
+
+ if (!kbdev)
+ {
+ return -ENODEV;
+ }
+
+ current_policy = kbase_pm_get_policy(kbdev);
+
+ policy_count = kbase_pm_list_policies(&policy_list);
+
+ for(i=0; i<policy_count && ret<PAGE_SIZE; i++)
+ {
+ if (policy_list[i] == current_policy)
+ {
+ ret += scnprintf(buf+ret, PAGE_SIZE - ret, "[%s] ", policy_list[i]->name);
+ }
+ else
+ {
+ ret += scnprintf(buf+ret, PAGE_SIZE - ret, "%s ", policy_list[i]->name);
+ }
+ }
+
+ if (ret < PAGE_SIZE - 1)
+ {
+ ret += scnprintf(buf+ret, PAGE_SIZE-ret, "\n");
+ }
+ else
+ {
+ buf[PAGE_SIZE-2] = '\n';
+ buf[PAGE_SIZE-1] = '\0';
+ ret = PAGE_SIZE-1;
+ }
+
+ return ret;
+}
+
+/** Store callback for the @c power_policy sysfs file.
+ *
+ * This function is called when the @c power_policy sysfs file is written to.
+ * It matches the requested policy against the available policies and if a
+ * matching policy is found calls @ref kbase_pm_set_policy to change the
+ * policy.
+ *
+ * @param dev The device with sysfs file is for
+ * @param attr The attributes of the sysfs file
+ * @param buf The value written to the sysfs file
+ * @param count The number of bytes written to the sysfs file
+ *
+ * @return @c count if the function succeeded. An error code on failure.
+ */
+static ssize_t set_policy(struct device *dev, struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct kbase_device *kbdev;
+ const struct kbase_pm_policy *new_policy = NULL;
+ const struct kbase_pm_policy * const *policy_list;
+ int policy_count;
+ int i;
+
+ kbdev = to_kbase_device(dev);
+
+ if (!kbdev)
+ {
+ return -ENODEV;
+ }
+
+ policy_count = kbase_pm_list_policies(&policy_list);
+
+ for(i=0; i<policy_count; i++)
+ {
+ if (sysfs_streq(policy_list[i]->name, buf))
+ {
+ new_policy = policy_list[i];
+ break;
+ }
+ }
+
+ if (!new_policy)
+ {
+ dev_err(dev, "power_policy: policy not found\n");
+ return -EINVAL;
+ }
+
+ kbase_pm_set_policy(kbdev, new_policy);
+ return count;
+}
+
+/** The sysfs file @c power_policy.
+ *
+ * This is used for obtaining information about the available policies,
+ * determining which policy is currently active, and changing the active
+ * policy.
+ */
+DEVICE_ATTR(power_policy, S_IRUGO|S_IWUSR, show_policy, set_policy);
+
+#if MALI_LICENSE_IS_GPL && (MALI_CUSTOMER_RELEASE == 0)
+/** Store callback for the @c js_timeouts sysfs file.
+ *
+ * This function is called to get the contents of the @c js_timeouts sysfs
+ * file. This file contains five values separated by whitespace. The values
+ * are basically the same as KBASE_CONFIG_ATTR_JS_SOFT_STOP_TICKS,
+ * KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS, KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS,
+ * KBASE_CONFIG_ATTR_JS_RESET_TICKS_SS, BASE_CONFIG_ATTR_JS_RESET_TICKS_NSS
+ * configuration values (in that order), with the difference that the js_timeout
+ * valus are expressed in MILLISECONDS.
+ *
+ * The js_timeouts sysfile file allows the current values in
+ * use by the job scheduler to get override. Note that a value needs to
+ * be other than 0 for it to override the current job scheduler value.
+ *
+ * @param dev The device with sysfs file is for
+ * @param attr The attributes of the sysfs file
+ * @param buf The value written to the sysfs file
+ * @param count The number of bytes written to the sysfs file
+ *
+ * @return @c count if the function succeeded. An error code on failure.
+ */
+static ssize_t set_js_timeouts(struct device *dev, struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct kbase_device *kbdev;
+ int items;
+ unsigned long js_soft_stop_ms;
+ unsigned long js_hard_stop_ms_ss;
+ unsigned long js_hard_stop_ms_nss;
+ unsigned long js_reset_ms_ss;
+ unsigned long js_reset_ms_nss;
+
+ kbdev = to_kbase_device(dev);
+ if (!kbdev)
+ {
+ return -ENODEV;
+ }
+
+ items = sscanf(buf, "%lu %lu %lu %lu %lu", &js_soft_stop_ms, &js_hard_stop_ms_ss, &js_hard_stop_ms_nss, &js_reset_ms_ss, &js_reset_ms_nss);
+ if (items == 5)
+ {
+ u64 ticks;
+
+ ticks = js_soft_stop_ms * 1000000ULL;
+ osk_divmod6432(&ticks, kbdev->js_data.scheduling_tick_ns);
+ kbdev->js_soft_stop_ticks = ticks;
+
+ ticks = js_hard_stop_ms_ss * 1000000ULL;
+ osk_divmod6432(&ticks, kbdev->js_data.scheduling_tick_ns);
+ kbdev->js_hard_stop_ticks_ss = ticks;
+
+ ticks = js_hard_stop_ms_nss * 1000000ULL;
+ osk_divmod6432(&ticks, kbdev->js_data.scheduling_tick_ns);
+ kbdev->js_hard_stop_ticks_nss = ticks;
+
+ ticks = js_reset_ms_ss * 1000000ULL;
+ osk_divmod6432(&ticks, kbdev->js_data.scheduling_tick_ns);
+ kbdev->js_reset_ticks_ss = ticks;
+
+ ticks = js_reset_ms_nss * 1000000ULL;
+ osk_divmod6432(&ticks, kbdev->js_data.scheduling_tick_ns);
+ kbdev->js_reset_ticks_nss = ticks;
+
+ dev_info(kbdev->osdev.dev, "Overriding KBASE_CONFIG_ATTR_JS_SOFT_STOP_TICKS with %lu ticks (%lu ms)\n", (unsigned long)kbdev->js_soft_stop_ticks, js_soft_stop_ms);
+ dev_info(kbdev->osdev.dev, "Overriding KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS with %lu ticks (%lu ms)\n", (unsigned long)kbdev->js_hard_stop_ticks_ss, js_hard_stop_ms_ss);
+ dev_info(kbdev->osdev.dev, "Overriding KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS with %lu ticks (%lu ms)\n", (unsigned long)kbdev->js_hard_stop_ticks_nss, js_hard_stop_ms_nss);
+ dev_info(kbdev->osdev.dev, "Overriding KBASE_CONFIG_ATTR_JS_RESET_TICKS_SS with %lu ticks (%lu ms)\n", (unsigned long)kbdev->js_reset_ticks_ss, js_reset_ms_ss);
+ dev_info(kbdev->osdev.dev, "Overriding KBASE_CONFIG_ATTR_JS_RESET_TICKS_NSS with %lu ticks (%lu ms)\n", (unsigned long)kbdev->js_reset_ticks_nss, js_reset_ms_nss);
+
+ return count;
+ }
+ else
+ {
+ dev_err(kbdev->osdev.dev, "Couldn't process js_timeouts write operation.\nUse format "
+ "<soft_stop_ms> <hard_stop_ms_ss> <hard_stop_ms_nss> <reset_ms_ss> <reset_ms_nss>\n");
+ return -EINVAL;
+ }
+}
+
+/** Show callback for the @c js_timeouts sysfs file.
+ *
+ * This function is called to get the contents of the @c js_timeouts sysfs
+ * file. It returns the last set values written to the js_timeouts sysfs file.
+ * If the file didn't get written yet, the values will be 0.
+ *
+ * @param dev The device this sysfs file is for
+ * @param attr The attributes of the sysfs file
+ * @param buf The output buffer for the sysfs file contents
+ *
+ * @return The number of bytes output to @c buf.
+ */
+static ssize_t show_js_timeouts(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct kbase_device *kbdev;
+ ssize_t ret;
+ u64 ms;
+ unsigned long js_soft_stop_ms;
+ unsigned long js_hard_stop_ms_ss;
+ unsigned long js_hard_stop_ms_nss;
+ unsigned long js_reset_ms_ss;
+ unsigned long js_reset_ms_nss;
+
+ kbdev = to_kbase_device(dev);
+ if (!kbdev)
+ {
+ return -ENODEV;
+ }
+
+ ms = (u64)kbdev->js_soft_stop_ticks * kbdev->js_data.scheduling_tick_ns;
+ osk_divmod6432(&ms, 1000000UL);
+ js_soft_stop_ms = (unsigned long)ms;
+
+ ms = (u64)kbdev->js_hard_stop_ticks_ss * kbdev->js_data.scheduling_tick_ns;
+ osk_divmod6432(&ms, 1000000UL);
+ js_hard_stop_ms_ss = (unsigned long)ms;
+
+ ms = (u64)kbdev->js_hard_stop_ticks_nss * kbdev->js_data.scheduling_tick_ns;
+ osk_divmod6432(&ms, 1000000UL);
+ js_hard_stop_ms_nss = (unsigned long)ms;
+
+ ms = (u64)kbdev->js_reset_ticks_ss * kbdev->js_data.scheduling_tick_ns;
+ osk_divmod6432(&ms, 1000000UL);
+ js_reset_ms_ss = (unsigned long)ms;
+
+ ms = (u64)kbdev->js_reset_ticks_nss * kbdev->js_data.scheduling_tick_ns;
+ osk_divmod6432(&ms, 1000000UL);
+ js_reset_ms_nss = (unsigned long)ms;
+
+ ret = scnprintf(buf, PAGE_SIZE, "%lu %lu %lu %lu %lu\n", js_soft_stop_ms, js_hard_stop_ms_ss, js_hard_stop_ms_nss, js_reset_ms_ss, js_reset_ms_nss);
+
+ if (ret >= PAGE_SIZE )
+ {
+ buf[PAGE_SIZE-2] = '\n';
+ buf[PAGE_SIZE-1] = '\0';
+ ret = PAGE_SIZE-1;
+ }
+
+ return ret;
+}
+/** The sysfs file @c js_timeouts.
+ *
+ * This is used to override the current job scheduler values for
+ * KBASE_CONFIG_ATTR_JS_STOP_STOP_TICKS_SS
+ * KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_SS
+ * KBASE_CONFIG_ATTR_JS_HARD_STOP_TICKS_NSS
+ * KBASE_CONFIG_ATTR_JS_RESET_TICKS_SS
+ * KBASE_CONFIG_ATTR_JS_RESET_TICKS_NSS.
+ */
+DEVICE_ATTR(js_timeouts, S_IRUGO|S_IWUSR, show_js_timeouts, set_js_timeouts);
+#endif /* MALI_LICENSE_IS_GPL && (MALI_CUSTOMER_RELEASE == 0) */
+
+#if MALI_DEBUG
+static ssize_t set_js_softstop_always(struct device *dev, struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct kbase_device *kbdev;
+ int items;
+ int softstop_always;
+
+ kbdev = to_kbase_device(dev);
+ if (!kbdev)
+ {
+ return -ENODEV;
+ }
+
+ items = sscanf(buf, "%d", &softstop_always);
+ if ((items == 1) && ((softstop_always == 0) || (softstop_always == 1)))
+ {
+ kbdev->js_data.softstop_always = (mali_bool) softstop_always;
+
+ dev_info(kbdev->osdev.dev, "Support for softstop on a single context: %s\n", (kbdev->js_data.softstop_always == MALI_FALSE)? "Disabled" : "Enabled");
+ return count;
+ }
+ else
+ {
+ dev_err(kbdev->osdev.dev, "Couldn't process js_softstop_always write operation.\nUse format "
+ "<soft_stop_always>\n");
+ return -EINVAL;
+ }
+}
+
+static ssize_t show_js_softstop_always(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct kbase_device *kbdev;
+ ssize_t ret;
+
+ kbdev = to_kbase_device(dev);
+ if (!kbdev)
+ {
+ return -ENODEV;
+ }
+
+ ret = scnprintf(buf, PAGE_SIZE, "%d\n", kbdev->js_data.softstop_always);
+
+ if (ret >= PAGE_SIZE )
+ {
+ buf[PAGE_SIZE-2] = '\n';
+ buf[PAGE_SIZE-1] = '\0';
+ ret = PAGE_SIZE-1;
+ }
+
+ return ret;
+}
+/**
+ * By default, soft-stops are disabled when only a single context is present. The ability to
+ * enable soft-stop when only a single context is present can be used for debug and unit-testing purposes.
+ * (see CL t6xx_stress_1 unit-test as an example whereby this feature is used.)
+ */
+DEVICE_ATTR(js_softstop_always, S_IRUGO|S_IWUSR, show_js_softstop_always, set_js_softstop_always);
+#endif /* MALI_DEBUG */
+
+#if MALI_DEBUG
+typedef void (kbasep_debug_command_func)( kbase_device * );
+
+typedef enum
+{
+ KBASEP_DEBUG_COMMAND_DUMPTRACE,
+
+ /* This must be the last enum */
+ KBASEP_DEBUG_COMMAND_COUNT
+} kbasep_debug_command_code;
+
+typedef struct kbasep_debug_command
+{
+ char *str;
+ kbasep_debug_command_func *func;
+} kbasep_debug_command;
+
+/** Debug commands supported by the driver */
+static const kbasep_debug_command debug_commands[] =
+{
+ {
+ .str = "dumptrace",
+ .func = &kbasep_trace_dump,
+ }
+};
+
+/** Show callback for the @c debug_command sysfs file.
+ *
+ * This function is called to get the contents of the @c debug_command sysfs
+ * file. This is a list of the available debug commands, separated by newlines.
+ *
+ * @param dev The device this sysfs file is for
+ * @param attr The attributes of the sysfs file
+ * @param buf The output buffer for the sysfs file contents
+ *
+ * @return The number of bytes output to @c buf.
+ */
+static ssize_t show_debug(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct kbase_device *kbdev;
+ int i;
+ ssize_t ret = 0;
+
+ kbdev = to_kbase_device(dev);
+
+ if (!kbdev)
+ {
+ return -ENODEV;
+ }
+
+ for(i=0; i<KBASEP_DEBUG_COMMAND_COUNT && ret<PAGE_SIZE; i++)
+ {
+ ret += scnprintf(buf+ret, PAGE_SIZE - ret, "%s\n", debug_commands[i].str);
+ }
+
+ if (ret >= PAGE_SIZE )
+ {
+ buf[PAGE_SIZE-2] = '\n';
+ buf[PAGE_SIZE-1] = '\0';
+ ret = PAGE_SIZE-1;
+ }
+
+ return ret;
+}
+
+/** Store callback for the @c debug_command sysfs file.
+ *
+ * This function is called when the @c debug_command sysfs file is written to.
+ * It matches the requested command against the available commands, and if
+ * a matching command is found calls the associated function from
+ * @ref debug_commands to issue the command.
+ *
+ * @param dev The device with sysfs file is for
+ * @param attr The attributes of the sysfs file
+ * @param buf The value written to the sysfs file
+ * @param count The number of bytes written to the sysfs file
+ *
+ * @return @c count if the function succeeded. An error code on failure.
+ */
+static ssize_t issue_debug(struct device *dev, struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct kbase_device *kbdev;
+ int i;
+
+ kbdev = to_kbase_device(dev);
+
+ if (!kbdev)
+ {
+ return -ENODEV;
+ }
+
+ for(i=0; i<KBASEP_DEBUG_COMMAND_COUNT; i++)
+ {
+ if (sysfs_streq(debug_commands[i].str, buf))
+ {
+ debug_commands[i].func( kbdev );
+ return count;
+ }
+ }
+
+ /* Debug Command not found */
+ dev_err(dev, "debug_command: command not known\n");
+ return -EINVAL;
+}
+
+/** The sysfs file @c debug_command.
+ *
+ * This is used to issue general debug commands to the device driver.
+ * Reading it will produce a list of debug commands, separated by newlines.
+ * Writing to it with one of those commands will issue said command.
+ */
+DEVICE_ATTR(debug_command, S_IRUGO|S_IWUSR, show_debug, issue_debug);
+#endif /*MALI_DEBUG*/
+#endif /*MALI_LICENSE_IS_GPL*/
+
+static int kbase_common_reg_map(kbase_device *kbdev)
+{
+ struct kbase_os_device *osdev = &kbdev->osdev;
+ int err = -ENOMEM;
+
+ osdev->reg_res = request_mem_region(osdev->reg_start,
+ osdev->reg_size,
+#if MALI_LICENSE_IS_GPL
+ dev_name(osdev->dev)
+#else
+ mali_dev_name
+#endif
+ );
+ if (!osdev->reg_res)
+ {
+ dev_err(osdev->dev, "Register window unavailable\n");
+ err = -EIO;
+ goto out_region;
+ }
+
+ osdev->reg = ioremap(osdev->reg_start, osdev->reg_size);
+ if (!osdev->reg)
+ {
+ dev_err(osdev->dev, "Can't remap register window\n");
+ err = -EINVAL;
+ goto out_ioremap;
+ }
+
+ return 0;
+
+out_ioremap:
+ release_resource(osdev->reg_res);
+ kfree(osdev->reg_res);
+out_region:
+ return err;
+}
+
+static void kbase_common_reg_unmap(kbase_device *kbdev)
+{
+ struct kbase_os_device *osdev = &kbdev->osdev;
+
+ iounmap(osdev->reg);
+ release_resource(osdev->reg_res);
+ kfree(osdev->reg_res);
+}
+
+static int kbase_common_device_init(kbase_device *kbdev)
+{
+ struct kbase_os_device *osdev = &kbdev->osdev;
+ int err = -ENOMEM;
+ mali_error mali_err;
+ enum
+ {
+ inited_mem = (1u << 0),
+ inited_job_slot = (1u << 1),
+ inited_pm = (1u << 2),
+ inited_js = (1u << 3),
+ inited_irqs = (1u << 4)
+#if MALI_LICENSE_IS_GPL
+ ,inited_debug = (1u << 5)
+ ,inited_js_softstop = (1u << 6)
+#endif
+#if MALI_CUSTOMER_RELEASE == 0
+ ,inited_js_timeouts = (1u << 7)
+#endif
+ /* BASE_HW_ISSUE_8401 */
+ ,inited_workaround = (1u << 8)
+ };
+
+ int inited = 0;
+
+#if MALI_LICENSE_IS_GPL
+ dev_set_drvdata(osdev->dev, kbdev);
+
+ osdev->mdev.minor = MISC_DYNAMIC_MINOR;
+ osdev->mdev.name = osdev->devname;
+ osdev->mdev.fops = &kbase_fops;
+ osdev->mdev.parent = get_device(osdev->dev);
+#endif
+
+ scnprintf(osdev->devname, DEVNAME_SIZE, "%s%d", kbase_drv_name, kbase_dev_nr++);
+
+#if MALI_LICENSE_IS_GPL
+ if (misc_register(&osdev->mdev))
+ {
+ dev_err(osdev->dev, "Couldn't register misc dev %s\n", osdev->devname);
+ err = -EINVAL;
+ goto out_misc;
+ }
+
+ if (device_create_file(osdev->dev, &dev_attr_power_policy))
+ {
+ dev_err(osdev->dev, "Couldn't create power_policy sysfs file\n");
+ goto out_file;
+ }
+
+ down(&kbase_dev_list_lock);
+ list_add(&osdev->entry, &kbase_dev_list);
+ up(&kbase_dev_list_lock);
+ dev_info(osdev->dev, "Probed as %s\n", dev_name(osdev->mdev.this_device));
+#endif
+
+ mali_err = kbase_pm_init(kbdev);
+ if (MALI_ERROR_NONE != mali_err)
+ {
+ goto out_partial;
+ }
+ inited |= inited_pm;
+
+ mali_err = kbase_mem_init(kbdev);
+ if (MALI_ERROR_NONE != mali_err)
+ {
+ goto out_partial;
+ }
+ inited |= inited_mem;
+
+ mali_err = kbase_job_slot_init(kbdev);
+ if (MALI_ERROR_NONE != mali_err)
+ {
+ goto out_partial;
+ }
+ inited |= inited_job_slot;
+
+ mali_err = kbasep_js_devdata_init(kbdev);
+ if (MALI_ERROR_NONE != mali_err)
+ {
+ goto out_partial;
+ }
+ inited |= inited_js;
+
+ err = kbase_install_interrupts(kbdev);
+ if (err)
+ {
+ goto out_partial;
+ }
+ inited |= inited_irqs;
+
+#if MALI_LICENSE_IS_GPL
+#if MALI_DEBUG
+ if (device_create_file(osdev->dev, &dev_attr_debug_command))
+ {
+ dev_err(osdev->dev, "Couldn't create debug_command sysfs file\n");
+ goto out_partial;
+ }
+ inited |= inited_debug;
+
+ if (device_create_file(osdev->dev, &dev_attr_js_softstop_always))
+ {
+ dev_err(osdev->dev, "Couldn't create js_softstop_always sysfs file\n");
+ goto out_partial;
+ }
+ inited |= inited_js_softstop;
+#endif /* MALI_DEBUG */
+#if MALI_CUSTOMER_RELEASE == 0
+ if (device_create_file(osdev->dev, &dev_attr_js_timeouts))
+ {
+ dev_err(osdev->dev, "Couldn't create js_timeouts sysfs file\n");
+ goto out_partial;
+ }
+ inited |= inited_js_timeouts;
+#endif /* MALI_CUSTOMER_RELEASE */
+#endif /*MALI_LICENSE_IS_GPL*/
+
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8401))
+ {
+ if (MALI_ERROR_NONE != kbasep_8401_workaround_init(kbdev))
+ {
+ goto out_partial;
+ }
+ inited |= inited_workaround;
+ }
+
+ mali_err = kbase_pm_powerup(kbdev);
+ if (MALI_ERROR_NONE == mali_err)
+ {
+ return 0;
+ }
+
+out_partial:
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8401))
+ {
+ if (inited & inited_workaround)
+ {
+ kbasep_8401_workaround_term(kbdev);
+ }
+ }
+
+#if MALI_LICENSE_IS_GPL
+#if MALI_CUSTOMER_RELEASE == 0
+ if (inited & inited_js_timeouts)
+ {
+ device_remove_file(kbdev->osdev.dev, &dev_attr_js_timeouts);
+ }
+#endif
+#if MALI_DEBUG
+ if (inited & inited_js_softstop)
+ {
+ device_remove_file(kbdev->osdev.dev, &dev_attr_js_softstop_always);
+ }
+
+ if (inited & inited_debug)
+ {
+ device_remove_file(kbdev->osdev.dev, &dev_attr_debug_command);
+ }
+#endif
+#endif /*MALI_LICENSE_IS_GPL*/
+
+ if (inited & inited_js)
+ {
+ kbasep_js_devdata_halt(kbdev);
+ }
+ if (inited & inited_job_slot)
+ {
+ kbase_job_slot_halt(kbdev);
+ }
+ if (inited & inited_mem)
+ {
+ kbase_mem_halt(kbdev);
+ }
+ if (inited & inited_pm)
+ {
+ kbase_pm_halt(kbdev);
+ }
+
+ if (inited & inited_irqs)
+ {
+ kbase_release_interrupts(kbdev);
+ }
+
+ if (inited & inited_js)
+ {
+ kbasep_js_devdata_term(kbdev);
+ }
+ if (inited & inited_job_slot)
+ {
+ kbase_job_slot_term(kbdev);
+ }
+ if (inited & inited_mem)
+ {
+ kbase_mem_term(kbdev);
+ }
+ if (inited & inited_pm)
+ {
+ kbase_pm_term(kbdev);
+ }
+
+#if MALI_LICENSE_IS_GPL
+ down(&kbase_dev_list_lock);
+ list_del(&osdev->entry);
+ up(&kbase_dev_list_lock);
+
+ device_remove_file(kbdev->osdev.dev, &dev_attr_power_policy);
+out_file:
+ misc_deregister(&kbdev->osdev.mdev);
+out_misc:
+ put_device(osdev->dev);
+#endif
+ return err;
+}
+
+#if MALI_LICENSE_IS_GPL
+static int kbase_platform_device_probe(struct platform_device *pdev)
+{
+ struct kbase_device *kbdev;
+ kbase_device_info *dev_info;
+ struct kbase_os_device *osdev;
+ struct resource *reg_res;
+ kbase_attribute *platform_data;
+ int err;
+ int i;
+ struct mali_base_gpu_core_props *core_props;
+#if MALI_NO_MALI
+ mali_error mali_err;
+#endif
+
+ dev_info = (kbase_device_info *)pdev->id_entry->driver_data;
+ kbdev = kbase_device_alloc();
+ if (!kbdev)
+ {
+ dev_err(&pdev->dev, "Can't allocate device\n");
+ err = -ENOMEM;
+ goto out;
+ }
+
+#if MALI_NO_MALI
+ mali_err = midg_device_create(kbdev);
+ if (MALI_ERROR_NONE != mali_err)
+ {
+ dev_err(&pdev->dev, "Can't initialize dummy model\n");
+ err = -ENOMEM;
+ goto out_midg;
+ }
+#endif
+
+ osdev = &kbdev->osdev;
+ osdev->dev = &pdev->dev;
+ platform_data = (kbase_attribute *)osdev->dev->platform_data;
+
+ if (NULL == platform_data)
+ {
+ dev_err(osdev->dev, "Platform data not specified\n");
+ err = -ENOENT;
+ goto out_free_dev;
+ }
+
+ if (MALI_TRUE != kbasep_validate_configuration_attributes(kbdev, platform_data))
+ {
+ dev_err(osdev->dev, "Configuration attributes failed to validate\n");
+ err = -EINVAL;
+ goto out_free_dev;
+ }
+ kbdev->config_attributes = platform_data;
+
+ /* 3 IRQ resources */
+ for (i = 0; i < 3; i++)
+ {
+ struct resource *irq_res;
+
+ irq_res = platform_get_resource(pdev, IORESOURCE_IRQ, i);
+ if (!irq_res)
+ {
+ dev_err(osdev->dev, "No IRQ resource at index %d\n", i);
+ err = -ENOENT;
+ goto out_free_dev;
+ }
+
+ osdev->irqs[i].irq = irq_res->start;
+ osdev->irqs[i].flags = (irq_res->flags & IRQF_TRIGGER_MASK);
+ }
+
+ /* the first memory resource is the physical address of the GPU registers */
+ reg_res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!reg_res)
+ {
+ dev_err(&pdev->dev, "Invalid register resource\n");
+ err = -ENOENT;
+ goto out_free_dev;
+ }
+
+ osdev->reg_start = reg_res->start;
+ osdev->reg_size = resource_size(reg_res);
+
+ err = kbase_common_reg_map(kbdev);
+ if (err)
+ {
+ goto out_free_dev;
+ }
+
+ if (MALI_ERROR_NONE != kbase_device_init(kbdev, dev_info))
+ {
+ dev_err(&pdev->dev, "Can't initialize device\n");
+ err = -ENOMEM;
+ goto out_reg_unmap;
+ }
+
+#if MALI_USE_UMP == 1
+ kbdev->memdev.ump_device_id = kbasep_get_config_value(kbdev, platform_data,
+ KBASE_CONFIG_ATTR_UMP_DEVICE);
+#endif /* MALI_USE_UMP == 1 */
+
+ kbdev->memdev.per_process_memory_limit = kbasep_get_config_value(kbdev, platform_data,
+ KBASE_CONFIG_ATTR_MEMORY_PER_PROCESS_LIMIT);
+
+ /* obtain min/max configured gpu frequencies */
+ core_props = &(kbdev->gpu_props.props.core_props);
+ core_props->gpu_freq_khz_min = kbasep_get_config_value(kbdev, platform_data,
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MIN);
+ core_props->gpu_freq_khz_max = kbasep_get_config_value(kbdev, platform_data,
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MAX);
+ kbdev->gpu_props.irq_throttle_time_us = kbasep_get_config_value(kbdev, platform_data,
+ KBASE_CONFIG_ATTR_GPU_IRQ_THROTTLE_TIME_US);
+
+ err = kbase_register_memory_regions(kbdev, (kbase_attribute *)osdev->dev->platform_data);
+ if (err)
+ {
+ dev_err(osdev->dev, "Failed to register memory regions\n");
+ goto out_term_dev;
+ }
+
+ err = kbase_common_device_init(kbdev);
+ if (err)
+ {
+ dev_err(osdev->dev, "Failed kbase_common_device_init\n");
+ goto out_term_dev;
+ }
+ return 0;
+
+out_term_dev:
+ kbase_device_term(kbdev);
+out_reg_unmap:
+ kbase_common_reg_unmap(kbdev);
+out_free_dev:
+#if MALI_NO_MALI
+ midg_device_destroy(kbdev);
+out_midg:
+#endif /* MALI_NO_MALI */
+ kbase_device_free(kbdev);
+out:
+ return err;
+}
+#endif /* MALI_LICENSE_IS_GPL */
+
+static int kbase_common_device_remove(struct kbase_device *kbdev)
+{
+ if (kbase_hw_has_issue(kbdev, BASE_HW_ISSUE_8401))
+ {
+ kbasep_8401_workaround_term(kbdev);
+ }
+#if MALI_LICENSE_IS_GPL
+ /* Remove the sys power policy file */
+ device_remove_file(kbdev->osdev.dev, &dev_attr_power_policy);
+#if MALI_DEBUG
+ device_remove_file(kbdev->osdev.dev, &dev_attr_js_softstop_always);
+ device_remove_file(kbdev->osdev.dev, &dev_attr_debug_command);
+#endif
+#endif
+
+ kbasep_js_devdata_halt(kbdev);
+ kbase_job_slot_halt(kbdev);
+ kbase_mem_halt(kbdev);
+ kbase_pm_halt(kbdev);
+
+ kbase_release_interrupts(kbdev);
+
+ kbasep_js_devdata_term(kbdev);
+ kbase_job_slot_term(kbdev);
+ kbase_mem_term(kbdev);
+ kbase_pm_term(kbdev);
+
+#if MALI_LICENSE_IS_GPL
+ down(&kbase_dev_list_lock);
+ list_del(&kbdev->osdev.entry);
+ up(&kbase_dev_list_lock);
+ misc_deregister(&kbdev->osdev.mdev);
+ put_device(kbdev->osdev.dev);
+#endif
+ kbase_common_reg_unmap(kbdev);
+ kbase_device_term(kbdev);
+#if MALI_NO_MALI
+ midg_device_destroy(kbdev);
+#endif /* MALI_NO_MALI */
+ kbase_device_free(kbdev);
+
+ return 0;
+}
+
+
+#if MALI_LICENSE_IS_GPL
+static int kbase_platform_device_remove(struct platform_device *pdev)
+{
+ struct kbase_device *kbdev = to_kbase_device(&pdev->dev);
+
+ if (!kbdev)
+ {
+ return -ENODEV;
+ }
+
+ return kbase_common_device_remove(kbdev);
+}
+
+/** Suspend callback from the OS.
+ *
+ * This is called by Linux when the device should suspend.
+ *
+ * @param dev The device to suspend
+ *
+ * @return A standard Linux error code
+ */
+static int kbase_device_suspend(struct device *dev)
+{
+ struct kbase_device *kbdev = to_kbase_device(dev);
+
+ if (!kbdev)
+ {
+ return -ENODEV;
+ }
+
+ /* Send the event to the power policy */
+ kbase_pm_send_event(kbdev, KBASE_PM_EVENT_SYSTEM_SUSPEND);
+
+ /* Wait for the policy to suspend the device */
+ kbase_pm_wait_for_power_down(kbdev);
+
+ return 0;
+}
+
+/** Resume callback from the OS.
+ *
+ * This is called by Linux when the device should resume from suspension.
+ *
+ * @param dev The device to resume
+ *
+ * @return A standard Linux error code
+ */
+static int kbase_device_resume(struct device *dev)
+{
+ struct kbase_device *kbdev = to_kbase_device(dev);
+
+ if (!kbdev)
+ {
+ return -ENODEV;
+ }
+
+ /* Send the event to the power policy */
+ kbase_pm_send_event(kbdev, KBASE_PM_EVENT_SYSTEM_RESUME);
+
+ /* Wait for the policy to resume the device */
+ kbase_pm_wait_for_power_up(kbdev);
+
+ return 0;
+}
+
+#define kbdev_info(x) ((kernel_ulong_t)&kbase_dev_info[(x)])
+
+static struct platform_device_id kbase_platform_id_table[] =
+{
+ {
+ .name = "mali-t6xm",
+ .driver_data = kbdev_info(KBASE_MALI_T6XM),
+ },
+ {
+ .name = "mali-t6f1",
+ .driver_data = kbdev_info(KBASE_MALI_T6F1),
+ },
+ {
+ .name = "mali-t601",
+ .driver_data = kbdev_info(KBASE_MALI_T601),
+ },
+ {
+ .name = "mali-t604",
+ .driver_data = kbdev_info(KBASE_MALI_T604),
+ },
+ {
+ .name = "mali-t608",
+ .driver_data = kbdev_info(KBASE_MALI_T608),
+ },
+ {},
+};
+
+MODULE_DEVICE_TABLE(platform, kbase_platform_id_table);
+
+/** The power management operations for the platform driver.
+ */
+static struct dev_pm_ops kbase_pm_ops =
+{
+ .suspend = kbase_device_suspend,
+ .resume = kbase_device_resume,
+};
+
+static struct platform_driver kbase_platform_driver =
+{
+ .probe = kbase_platform_device_probe,
+ .remove = kbase_platform_device_remove,
+ .driver =
+ {
+ .name = kbase_drv_name,
+ .owner = THIS_MODULE,
+ .pm = &kbase_pm_ops,
+ },
+ .id_table = kbase_platform_id_table,
+};
+
+#endif /* MALI_LICENSE_IS_GPL */
+
+#if MALI_LICENSE_IS_GPL && MALI_FAKE_PLATFORM_DEVICE
+static struct platform_device *mali_device;
+#endif /* MALI_LICENSE_IS_GPL && MALI_FAKE_PLATFORM_DEVICE */
+
+#ifdef MALI_PCI_DEVICE
+static kbase_attribute pci_attributes[] =
+{
+ {
+ KBASE_CONFIG_ATTR_MEMORY_PER_PROCESS_LIMIT,
+ 512 * 1024 * 1024UL /* 512MB */
+ },
+#if MALI_USE_UMP == 1
+ {
+ KBASE_CONFIG_ATTR_UMP_DEVICE,
+ UMP_DEVICE_Z_SHIFT
+ },
+#endif /* MALI_USE_UMP == 1 */
+ {
+ KBASE_CONFIG_ATTR_MEMORY_OS_SHARED_MAX,
+ 768 * 1024 * 1024UL /* 768MB */
+ },
+ {
+ KBASE_CONFIG_ATTR_END,
+ 0
+ }
+};
+
+static int kbase_pci_device_probe(struct pci_dev *pdev,
+ const struct pci_device_id *pci_id)
+{
+ const kbase_device_info *dev_info;
+ kbase_device *kbdev;
+ kbase_os_device *osdev;
+ kbase_attribute *platform_data;
+ struct mali_base_gpu_core_props *core_props;
+ int err;
+#if MALI_NO_MALI
+ mali_error mali_err;
+#endif
+
+ dev_info = &kbase_dev_info[pci_id->driver_data];
+ kbdev = kbase_device_alloc();
+ if (!kbdev)
+ {
+ dev_err(&pdev->dev, "Can't allocate device\n");
+ err = -ENOMEM;
+ goto out;
+ }
+
+#if MALI_NO_MALI
+ mali_err = midg_device_create(kbdev);
+ if (MALI_ERROR_NONE != mali_err)
+ {
+ dev_err(&pdev->dev, "Can't initialize dummy model\n");
+ err = -ENOMEM;
+ goto out_midg;
+ }
+#endif
+
+ osdev = &kbdev->osdev;
+ osdev->dev = &pdev->dev;
+ platform_data = (kbase_attribute *)osdev->dev->platform_data;
+
+ err = pci_enable_device(pdev);
+ if (err)
+ {
+ goto out_free_dev;
+ }
+
+ osdev->reg_start = pci_resource_start(pdev, 0);
+ osdev->reg_size = pci_resource_len(pdev, 0);
+ if (!(pci_resource_flags(pdev, 0) & IORESOURCE_MEM))
+ {
+ err = -EINVAL;
+ goto out_disable;
+ }
+
+ err = kbase_common_reg_map(kbdev);
+ if (err)
+ {
+ goto out_disable;
+ }
+
+ if (MALI_ERROR_NONE != kbase_device_init(kbdev, dev_info))
+ {
+ dev_err(&pdev->dev, "Can't initialize device\n");
+ err = -ENOMEM;
+ goto out_reg_unmap;
+ }
+
+ osdev->irqs[0].irq = pdev->irq;
+ osdev->irqs[1].irq = pdev->irq;
+ osdev->irqs[2].irq = pdev->irq;
+
+ pci_set_master(pdev);
+
+ if (MALI_TRUE != kbasep_validate_configuration_attributes(kbdev, pci_attributes))
+ {
+ err = -EINVAL;
+ goto out_term_dev;
+ }
+ /* Use the master passed in instead of the pci attributes */
+ kbdev->config_attributes = platform_data;
+
+#if MALI_USE_UMP == 1
+ kbdev->memdev.ump_device_id = kbasep_get_config_value(kbdev, pci_attributes,
+ KBASE_CONFIG_ATTR_UMP_DEVICE);
+#endif /* MALI_USE_UMP == 1 */
+
+ kbdev->memdev.per_process_memory_limit = kbasep_get_config_value(kbdev, pci_attributes,
+ KBASE_CONFIG_ATTR_MEMORY_PER_PROCESS_LIMIT);
+
+ err = kbase_register_memory_regions(kbdev, pci_attributes);
+ if (err)
+ {
+ goto out_term_dev;
+ }
+
+ /* obtain min/max configured gpu frequencies */
+ core_props = &(kbdev->gpu_props.props.core_props);
+ core_props->gpu_freq_khz_min = kbasep_get_config_value(kbdev, platform_data,
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MIN);
+ core_props->gpu_freq_khz_max = kbasep_get_config_value(kbdev, platform_data,
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MAX);
+ kbdev->gpu_props.irq_throttle_time_us = kbasep_get_config_value(kbdev, platform_data,
+ KBASE_CONFIG_ATTR_GPU_IRQ_THROTTLE_TIME_US);
+
+ err = kbase_common_device_init(kbdev);
+ if (err)
+ {
+ goto out_term_dev;
+ }
+
+ return 0;
+
+out_term_dev:
+ kbase_device_term(kbdev);
+out_reg_unmap:
+ kbase_common_reg_unmap(kbdev);
+out_disable:
+ pci_disable_device(pdev);
+out_free_dev:
+#if MALI_NO_MALI
+ midg_device_destroy(kbdev);
+out_midg:
+#endif /* MALI_NO_MALI */
+ kbase_device_free(kbdev);
+out:
+ return err;
+}
+
+static void kbase_pci_device_remove(struct pci_dev *pdev)
+{
+ struct kbase_device *kbdev = to_kbase_device(&pdev->dev);
+
+ if (!kbdev)
+ {
+ return;
+ }
+
+ kbase_common_device_remove(kbdev);
+ pci_disable_device(pdev);
+}
+
+static DEFINE_PCI_DEVICE_TABLE(kbase_pci_id_table) =
+{
+ { PCI_DEVICE(0x13b5, 0x6956), 0, 0, KBASE_MALI_T6XM },
+ {},
+};
+
+MODULE_DEVICE_TABLE(pci, kbase_pci_id_table);
+
+static struct pci_driver kbase_pci_driver =
+{
+ .name = KBASE_DRV_NAME,
+ .probe = kbase_pci_device_probe,
+ .remove = kbase_pci_device_remove,
+ .id_table = kbase_pci_id_table,
+};
+#endif /* MALI_PCI_DEVICE */
+
+#if MALI_LICENSE_IS_GPL
+static int __init kbase_driver_init(void)
+{
+ int err;
+#if MALI_FAKE_PLATFORM_DEVICE
+ kbase_platform_config *config;
+ int attribute_count;
+ struct resource resources[PLATFORM_CONFIG_RESOURCE_COUNT];
+
+ config = kbasep_get_platform_config();
+ attribute_count = kbasep_get_config_attribute_count(config->attributes);
+ mali_device = platform_device_alloc( kbasep_midgard_type_to_string(config->midgard_type), 0);
+ if (mali_device == NULL)
+ {
+ return -ENOMEM;
+ }
+
+ kbasep_config_parse_io_resources(config->io_resources, resources);
+ err = platform_device_add_resources(mali_device, resources, PLATFORM_CONFIG_RESOURCE_COUNT);
+ if (err)
+ {
+ platform_device_put(mali_device);
+ mali_device = NULL;
+ return err;
+ }
+
+ err = platform_device_add_data(mali_device, config->attributes, attribute_count * sizeof(config->attributes[0]));
+ if (err)
+ {
+ platform_device_unregister(mali_device);
+ mali_device = NULL;
+ return err;
+ }
+
+ err = platform_device_add(mali_device);
+ if (err)
+ {
+ platform_device_unregister(mali_device);
+ mali_device = NULL;
+ return err;
+ }
+
+#endif /* MALI_FAKE_PLATFORM_DEVICE */
+ err = platform_driver_register(&kbase_platform_driver);
+ if (err)
+ {
+ return err;
+ }
+
+#ifdef MALI_PCI_DEVICE
+ err = pci_register_driver(&kbase_pci_driver);
+ if (err)
+ {
+ platform_driver_unregister(&kbase_platform_driver);
+ return err;
+ }
+#endif
+
+ return 0;
+}
+#else
+static int __init kbase_driver_init(void)
+{
+ kbase_platform_config *config;
+ struct kbase_device *kbdev;
+ const kbase_device_info *dev_info;
+ kbase_os_device *osdev;
+ int err;
+ dev_t dev = 0;
+ struct mali_base_gpu_core_props *core_props;
+
+ if (0 == mali_major)
+ {
+ /* auto select a major */
+ err = alloc_chrdev_region(&dev, 0, 1, mali_dev_name);
+ mali_major = MAJOR(dev);
+ }
+ else
+ {
+ /* use load time defined major number */
+ dev = MKDEV(mali_major, 0);
+ err = register_chrdev_region(dev, 1, mali_dev_name);
+ }
+
+ if (0 != err)
+ {
+ goto out_region;
+ }
+
+ memset(&mali_linux_device, 0, sizeof(mali_linux_device));
+
+ /* initialize our char dev data */
+ cdev_init(&mali_linux_device.cdev, &kbase_fops);
+ mali_linux_device.cdev.owner = THIS_MODULE;
+ mali_linux_device.cdev.ops = &kbase_fops;
+
+ /* register char dev with the kernel */
+ err = cdev_add(&mali_linux_device.cdev, dev, 1/*count*/);
+ if (0 != err)
+ {
+ goto out_cdev_add;
+ }
+
+ config = kbasep_get_platform_config();
+
+ dev_info = &kbase_dev_info[config->midgard_type];
+ kbdev = kbase_device_alloc();
+ if (!kbdev)
+ {
+ dev_err(&pdev->dev, "Can't allocate device\n");
+ err = -ENOMEM;
+ goto out_kbdev_alloc;
+ }
+
+#if MALI_NO_MALI
+ mali_err = midg_device_create(kbdev);
+ if (MALI_ERROR_NONE != mali_err)
+ {
+ dev_err(&pdev->dev, "Can't initialize dummy model\n");
+ err = -ENOMEM;
+ goto out_midg;
+ }
+#endif
+
+ osdev = &kbdev->osdev;
+ osdev->dev = &mali_linux_device.cdev;
+ osdev->reg_start = config->io_resources->io_memory_region.start;
+ osdev->reg_size = config->io_resources->io_memory_region.end - config->io_resources->io_memory_region.start + 1;
+
+ err = kbase_common_reg_map(kbdev);
+ if (err)
+ {
+ goto out_free_dev;
+ }
+
+ if (MALI_ERROR_NONE != kbase_device_init(kbdev, dev_info))
+ {
+ dev_err(&pdev->dev, "Can't initialize device\n");
+ err = -ENOMEM;
+ goto out_reg_unmap;
+ }
+
+ if (MALI_TRUE != kbasep_validate_configuration_attributes(kbdev, config->attributes))
+ {
+ err = -EINVAL;
+ goto out_device_init;
+ }
+
+ kbdev->config_attributes = config->attributes;
+
+ osdev->irqs[0].irq = config->io_resources->job_irq_number;
+ osdev->irqs[1].irq = config->io_resources->mmu_irq_number;
+ osdev->irqs[2].irq = config->io_resources->gpu_irq_number;
+
+ kbdev->memdev.per_process_memory_limit = kbasep_get_config_value(kbdev, config->attributes,
+ KBASE_CONFIG_ATTR_MEMORY_PER_PROCESS_LIMIT);
+
+#if MALI_USE_UMP == 1
+ kbdev->memdev.ump_device_id = kbasep_get_config_value(kbdev, config->attributes, KBASE_CONFIG_ATTR_UMP_DEVICE);
+#endif /* MALI_USE_UMP == 1 */
+
+ /* obtain min/max configured gpu frequencies */
+ core_props = &(kbdev->gpu_props.props.core_props);
+ core_props->gpu_freq_khz_min = kbasep_get_config_value(kbdev, config->attributes,
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MIN);
+ core_props->gpu_freq_khz_max = kbasep_get_config_value(kbdev, config->attributes,
+ KBASE_CONFIG_ATTR_GPU_FREQ_KHZ_MAX);
+ kbdev->gpu_props.irq_throttle_time_us = kbasep_get_config_value(kbdev, config->attributes,
+ KBASE_CONFIG_ATTR_GPU_IRQ_THROTTLE_TIME_US);
+
+ err = kbase_register_memory_regions(kbdev, config->attributes);
+ if (err)
+ {
+ goto out_device_init;
+ }
+
+ err = kbase_common_device_init(kbdev);
+ if (0 != err)
+ {
+ goto out_device_init;
+ }
+
+ g_kbdev = kbdev;
+
+ return 0;
+
+out_device_init:
+ kbase_device_term(kbdev);
+ g_kbdev = NULL;
+out_reg_unmap:
+ kbase_common_reg_unmap(kbdev);
+out_free_dev:
+#if MALI_NO_MALI
+ midg_device_destroy(kbdev);
+out_midg:
+#endif /* MALI_NO_MALI */
+ kbase_device_free(kbdev);
+out_kbdev_alloc:
+ cdev_del(&mali_linux_device.cdev);
+out_cdev_add:
+ unregister_chrdev_region(dev, 1);
+out_region:
+ return err;
+}
+
+#endif /* MALI_LICENSE_IS_GPL */
+
+static void __exit kbase_driver_exit(void)
+{
+#if MALI_LICENSE_IS_GPL
+#ifdef MALI_PCI_DEVICE
+ pci_unregister_driver(&kbase_pci_driver);
+#endif
+ platform_driver_unregister(&kbase_platform_driver);
+#if MALI_FAKE_PLATFORM_DEVICE
+ if (mali_device)
+ {
+ platform_device_unregister(mali_device);
+ }
+#endif
+#else
+ dev_t dev = MKDEV(mali_major, 0);
+ struct kbase_device *kbdev = g_kbdev;
+
+ if (!kbdev)
+ {
+ return;
+ }
+
+ kbase_common_device_remove(kbdev);
+
+ /* unregister char device */
+ cdev_del(&mali_linux_device.cdev);
+
+ /* free major */
+ unregister_chrdev_region(dev, 1);
+#endif
+}
+
+module_init(kbase_driver_init);
+module_exit(kbase_driver_exit);
+
+#if MALI_LICENSE_IS_GPL
+MODULE_LICENSE("GPL");
+#else
+MODULE_LICENSE("Proprietary");
+#endif
+
+#if MALI_GATOR_SUPPORT
+/* Create the trace points (otherwise we just get code to call a tracepoint) */
+#define CREATE_TRACE_POINTS
+#include "mali_linux_trace.h"
+
+void kbase_trace_mali_pm_status(u32 event, u64 value)
+{
+ trace_mali_pm_status(event, value);
+}
+
+void kbase_trace_mali_pm_power_off(u32 event, u64 value)
+{
+ trace_mali_pm_power_off(event, value);
+}
+
+void kbase_trace_mali_pm_power_on(u32 event, u64 value)
+{
+ trace_mali_pm_power_on(event, value);
+}
+
+void kbase_trace_mali_job_slots_event(u32 event)
+{
+ trace_mali_job_slots_event(event);
+}
+
+void kbase_trace_mali_page_fault_insert_pages(int event, u32 value)
+{
+ trace_mali_page_fault_insert_pages(event, value);
+}
+
+void kbase_trace_mali_mmu_as_in_use(int event)
+{
+ trace_mali_mmu_as_in_use(event);
+}
+void kbase_trace_mali_mmu_as_released(int event)
+{
+ trace_mali_mmu_as_released(event);
+}
+void kbase_trace_mali_total_alloc_pages_change(long long int event)
+{
+ trace_mali_total_alloc_pages_change(event);
+}
+#endif
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_linux.h
+ * Base kernel APIs, Linux implementation.
+ */
+
+#ifndef _KBASE_LINUX_H_
+#define _KBASE_LINUX_H_
+
+/* All things that are needed for the Linux port. */
+#if MALI_LICENSE_IS_GPL
+#include <linux/platform_device.h>
+#include <linux/miscdevice.h>
+#endif
+#include <linux/list.h>
+#include <linux/module.h>
+
+typedef struct kbase_os_context
+{
+ u64 cookies;
+ osk_dlist reg_pending;
+ wait_queue_head_t event_queue;
+} kbase_os_context;
+
+
+#define DEVNAME_SIZE 16
+
+typedef struct kbase_os_device
+{
+#if MALI_LICENSE_IS_GPL
+ struct list_head entry;
+ struct device *dev;
+ struct miscdevice mdev;
+#else
+ struct cdev *dev;
+#endif
+ u64 reg_start;
+ size_t reg_size;
+ void __iomem *reg;
+ struct resource *reg_res;
+ struct {
+ int irq;
+ int flags;
+ } irqs[3];
+ char devname[DEVNAME_SIZE];
+
+#if MALI_NO_MALI
+ void *model;
+ struct kmem_cache *irq_slab;
+ osk_workq irq_workq;
+ osk_atomic serving_job_irq;
+ osk_atomic serving_gpu_irq;
+ osk_atomic serving_mmu_irq;
+ osk_spinlock reg_op_lock;
+#endif
+} kbase_os_device;
+
+#define KBASE_OS_SUPPORT 1
+
+#if defined(MALI_KERNEL_TEST_API)
+#if (1 == MALI_KERNEL_TEST_API)
+#define KBASE_EXPORT_TEST_API(func) EXPORT_SYMBOL(func);
+#else
+#define KBASE_EXPORT_TEST_API(func)
+#endif
+#else
+#define KBASE_EXPORT_TEST_API(func)
+#endif
+
+#define KBASE_EXPORT_SYMBOL(func) EXPORT_SYMBOL(func);
+
+#endif /* _KBASE_LINUX_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_mem_linux.c
+ * Base kernel memory APIs, Linux implementation.
+ */
+
+/* #define DEBUG 1 */
+
+#include <linux/kernel.h>
+#include <linux/bug.h>
+#include <linux/mm.h>
+#include <linux/fs.h>
+#include <linux/dma-mapping.h>
+
+#include <kbase/src/common/mali_kbase.h>
+#include <kbase/src/linux/mali_kbase_mem_linux.h>
+
+struct kbase_va_region *kbase_pmem_alloc(struct kbase_context *kctx, u32 size,
+ u32 flags, u16 *pmem_cookie)
+{
+ struct kbase_va_region *reg;
+ u16 cookie;
+
+ OSK_ASSERT(kctx != NULL);
+ OSK_ASSERT(pmem_cookie != NULL);
+
+ if ( 0 == size )
+ {
+ goto out1;
+ }
+
+ if (!kbase_check_alloc_flags(flags))
+ {
+ goto out1;
+ }
+
+ reg = kbase_alloc_free_region(kctx, 0, size, KBASE_REG_ZONE_PMEM);
+ if (!reg)
+ goto out1;
+
+ reg->flags &= ~KBASE_REG_FREE;
+
+ kbase_update_region_flags(reg, flags, MALI_FALSE);
+
+ if (kbase_alloc_phy_pages(reg, size, size))
+ goto out2;
+
+ reg->nr_alloc_pages = size;
+ reg->extent = 0;
+
+ kbase_gpu_vm_lock(kctx);
+ if (!kctx->osctx.cookies)
+ goto out3;
+
+ cookie = __ffs(kctx->osctx.cookies);
+ kctx->osctx.cookies &= ~(1UL << cookie);
+ reg->flags &= ~KBASE_REG_COOKIE_MASK;
+ reg->flags |= KBASE_REG_COOKIE(cookie);
+
+ OSK_DLIST_PUSH_FRONT(&kctx->osctx.reg_pending, reg,
+ struct kbase_va_region, link);
+
+ *pmem_cookie = cookie;
+ kbase_gpu_vm_unlock(kctx);
+
+ return reg;
+
+out3:
+ kbase_gpu_vm_unlock(kctx);
+ kbase_free_phy_pages(reg);
+out2:
+ osk_free(reg);
+out1:
+ return NULL;
+
+}
+KBASE_EXPORT_TEST_API(kbase_pmem_alloc)
+
+/*
+ * Callback for munmap(). PMEM receives a special treatment, as it
+ * frees the memory at the same time it gets unmapped. This avoids the
+ * map/unmap race where map reuses a memory range that has been
+ * unmapped from CPU, but still mapped on GPU.
+ */
+STATIC void kbase_cpu_vm_close(struct vm_area_struct *vma)
+{
+ struct kbase_va_region *reg = vma->vm_private_data;
+ kbase_context *kctx = reg->kctx;
+ mali_error err;
+
+ kbase_gpu_vm_lock(kctx);
+
+ err = kbase_cpu_free_mapping(reg, vma);
+ if (!err &&
+ (reg->flags & KBASE_REG_ZONE_MASK) == KBASE_REG_ZONE_PMEM)
+ {
+ kbase_mem_free_region(kctx, reg);
+ }
+
+ kbase_gpu_vm_unlock(kctx);
+}
+KBASE_EXPORT_TEST_API(kbase_cpu_vm_close)
+
+static const struct vm_operations_struct kbase_vm_ops = {
+ .close = kbase_cpu_vm_close,
+};
+
+static int kbase_cpu_mmap(struct kbase_va_region *reg, struct vm_area_struct *vma, void *kaddr, u32 nr_pages)
+{
+ struct kbase_cpu_mapping *map;
+ u64 start_off = vma->vm_pgoff - reg->start_pfn;
+ osk_phy_addr *page_array;
+ int err = 0;
+ int i;
+
+ map = osk_calloc(sizeof(*map));
+ if (!map)
+ {
+ WARN_ON(1);
+ err = -ENOMEM;
+ goto out;
+ }
+
+ /*
+ * VM_DONTCOPY - don't make this mapping available in fork'ed processes
+ * VM_DONTEXPAND - disable mremap on this region
+ * VM_RESERVED & VM_IO - disables paging
+ * VM_MIXEDMAP - Support mixing struct page*s and raw pfns.
+ * This is needed to support using the dedicated and
+ * the OS based memory backends together.
+ */
+ /*
+ * This will need updating to propagate coherency flags
+ * See MIDBASE-1057
+ */
+ vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND | VM_RESERVED | VM_IO | VM_MIXEDMAP;
+ vma->vm_ops = &kbase_vm_ops;
+ vma->vm_private_data = reg;
+
+ page_array = kbase_get_phy_pages(reg);
+
+ if (!(reg->flags & KBASE_REG_CPU_CACHED))
+ {
+ /* We can't map vmalloc'd memory uncached.
+ * Other memory will have been returned from
+ * osk_phy_pages_alloc which should have done the cache
+ * maintenance necessary to support an uncached mapping
+ */
+ BUG_ON(kaddr);
+ vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
+ }
+
+ if (!kaddr)
+ {
+ for (i = 0; i < nr_pages; i++)
+ {
+ err = vm_insert_mixed(vma, vma->vm_start + (i << OSK_PAGE_SHIFT), page_array[i + start_off] >> OSK_PAGE_SHIFT);
+ WARN_ON(err);
+ if (err)
+ break;
+ }
+ }
+ else
+ {
+ /* vmalloc remaping is easy... */
+ err = remap_vmalloc_range(vma, kaddr, 0);
+ WARN_ON(err);
+ }
+
+ if (err)
+ {
+ osk_free(map);
+ goto out;
+ }
+
+ map->uaddr = (osk_virt_addr)vma->vm_start;
+ map->nr_pages = nr_pages;
+ map->page_off = start_off;
+ map->private = vma;
+
+ OSK_DLIST_PUSH_FRONT(®->map_list, map,
+ struct kbase_cpu_mapping, link);
+
+out:
+ return err;
+}
+
+static int kbase_rb_mmap(struct kbase_context *kctx,
+ struct vm_area_struct *vma,
+ struct kbase_va_region **reg,
+ void **kmap_addr)
+{
+ struct kbase_va_region *new_reg;
+ void *kaddr;
+ u32 nr_pages;
+ size_t size;
+ int err = 0;
+ mali_error m_err = MALI_ERROR_NONE;
+
+ pr_debug("in kbase_rb_mmap\n");
+ size = (vma->vm_end - vma->vm_start);
+ nr_pages = size >> OSK_PAGE_SHIFT;
+
+ if (kctx->jctx.pool_size < size)
+ {
+ err = -EINVAL;
+ goto out;
+ }
+
+ kaddr = kctx->jctx.pool;
+
+ new_reg = kbase_alloc_free_region(kctx, 0, nr_pages, KBASE_REG_ZONE_PMEM);
+ if (!new_reg)
+ {
+ err = -ENOMEM;
+ WARN_ON(1);
+ goto out;
+ }
+
+ new_reg->flags &= ~KBASE_REG_FREE;
+ new_reg->flags |= KBASE_REG_IS_RB | KBASE_REG_CPU_CACHED;
+
+ m_err = kbase_add_va_region(kctx, new_reg, vma->vm_start, nr_pages, 1);
+ if (MALI_ERROR_NONE != m_err)
+ {
+ pr_debug("kbase_rb_mmap: kbase_add_va_region failed\n");
+ /* Free allocated new_reg */
+ kbase_free_alloced_region(new_reg);
+ err = -ENOMEM;
+ goto out;
+ }
+
+ *kmap_addr = kaddr;
+ *reg = new_reg;
+
+ pr_debug("kbase_rb_mmap done\n");
+ return 0;
+
+out:
+ return err;
+}
+
+static int kbase_trace_buffer_mmap(struct kbase_context * kctx, struct vm_area_struct * vma, struct kbase_va_region **reg, void **kaddr)
+{
+ struct kbase_va_region *new_reg;
+ u32 nr_pages;
+ size_t size;
+ int err = 0;
+ u32 * tb;
+
+ pr_debug("in %s\n", __func__);
+ size = (vma->vm_end - vma->vm_start);
+ nr_pages = size >> OSK_PAGE_SHIFT;
+
+ if (!kctx->jctx.tb)
+ {
+ tb = osk_vmalloc(size);
+ if (NULL == tb)
+ {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ kbase_device_trace_buffer_install(kctx, tb, size);
+ }
+ else
+ {
+ err = -EINVAL;
+ goto out;
+ }
+
+ *kaddr = kctx->jctx.tb;
+
+ new_reg = kbase_alloc_free_region(kctx, 0, nr_pages, KBASE_REG_ZONE_PMEM);
+ if (!new_reg)
+ {
+ err = -ENOMEM;
+ WARN_ON(1);
+ goto out_disconnect;
+ }
+
+ new_reg->flags &= ~KBASE_REG_FREE;
+ new_reg->flags |= KBASE_REG_IS_TB | KBASE_REG_CPU_CACHED;
+
+ if (MALI_ERROR_NONE != kbase_add_va_region(kctx, new_reg, vma->vm_start, nr_pages, 1))
+ {
+ err = -ENOMEM;
+ WARN_ON(1);
+ goto out_va_region;
+ }
+
+ *reg = new_reg;
+
+ /* map read only, noexec */
+ vma->vm_flags &= ~(VM_WRITE|VM_EXEC);
+ /* the rest of the flags is added by the cpu_mmap handler */
+
+ pr_debug("%s done\n", __func__);
+ return 0;
+
+out_va_region:
+ kbase_free_alloced_region(new_reg);
+out_disconnect:
+ kbase_device_trace_buffer_uninstall(kctx);
+ osk_vfree(tb);
+out:
+ return err;
+
+}
+
+static int kbase_mmu_dump_mmap( struct kbase_context *kctx,
+ struct vm_area_struct *vma,
+ struct kbase_va_region **reg,
+ void **kmap_addr )
+{
+ struct kbase_va_region *new_reg;
+ void *kaddr;
+ u32 nr_pages;
+ size_t size;
+ int err = 0;
+
+ pr_debug("in kbase_mmu_dump_mmap\n");
+ size = (vma->vm_end - vma->vm_start);
+ nr_pages = size >> OSK_PAGE_SHIFT;
+
+ kaddr = kbase_mmu_dump(kctx, nr_pages);
+
+ if (!kaddr)
+ {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ new_reg = kbase_alloc_free_region(kctx, 0, nr_pages, KBASE_REG_ZONE_PMEM);
+ if (!new_reg)
+ {
+ err = -ENOMEM;
+ WARN_ON(1);
+ goto out;
+ }
+
+ new_reg->flags &= ~KBASE_REG_FREE;
+ new_reg->flags |= KBASE_REG_IS_MMU_DUMP | KBASE_REG_CPU_CACHED;
+
+ if (MALI_ERROR_NONE != kbase_add_va_region(kctx, new_reg, vma->vm_start, nr_pages, 1))
+ {
+ err = -ENOMEM;
+ WARN_ON(1);
+ goto out_va_region;
+ }
+
+ *kmap_addr = kaddr;
+ *reg = new_reg;
+
+ pr_debug("kbase_mmu_dump_mmap done\n");
+ return 0;
+
+out_va_region:
+ kbase_free_alloced_region(new_reg);
+out:
+ return err;
+}
+
+/* must be called with the gpu vm lock held */
+
+struct kbase_va_region * kbase_lookup_cookie(struct kbase_context * kctx, mali_addr64 cookie)
+{
+ struct kbase_va_region * reg;
+ mali_addr64 test_cookie;
+
+ OSK_ASSERT(kctx != NULL);
+
+ test_cookie = KBASE_REG_COOKIE(cookie);
+
+ OSK_DLIST_FOREACH(&kctx->osctx.reg_pending, struct kbase_va_region, link, reg)
+ {
+ if ((reg->flags & KBASE_REG_COOKIE_MASK) == test_cookie)
+ {
+ return reg;
+ }
+ }
+
+ return NULL; /* not found */
+}
+KBASE_EXPORT_TEST_API(kbase_lookup_cookie)
+
+void kbase_unlink_cookie(struct kbase_context * kctx, mali_addr64 cookie, struct kbase_va_region * reg)
+{
+ OSKP_ASSERT(kctx != NULL);
+ OSKP_ASSERT(reg != NULL);
+ OSKP_ASSERT(MALI_TRUE == OSK_DLIST_MEMBER_OF(&kctx->osctx.reg_pending, reg, link));
+ OSKP_ASSERT(KBASE_REG_COOKIE(cookie) == (reg->flags & KBASE_REG_COOKIE_MASK));
+ OSKP_ASSERT((kctx->osctx.cookies & (1UL << cookie)) == 0);
+
+ OSK_DLIST_REMOVE(&kctx->osctx.reg_pending, reg, link);
+ kctx->osctx.cookies |= (1UL << cookie); /* mark as resolved */
+}
+
+KBASE_EXPORT_TEST_API(kbase_unlink_cookie)
+
+void kbase_os_mem_map_lock(struct kbase_context * kctx)
+{
+ struct mm_struct * mm = current->mm;
+ (void)kctx;
+ down_read(&mm->mmap_sem);
+}
+
+void kbase_os_mem_map_unlock(struct kbase_context * kctx)
+{
+ struct mm_struct * mm = current->mm;
+ (void)kctx;
+ up_read(&mm->mmap_sem);
+}
+
+int kbase_mmap(struct file *file, struct vm_area_struct *vma)
+{
+ struct kbase_context *kctx = file->private_data;
+ struct kbase_va_region *reg;
+ void *kaddr = NULL;
+ u32 nr_pages;
+ int err = 0;
+
+ pr_debug("kbase_mmap\n");
+ nr_pages = (vma->vm_end - vma->vm_start) >> OSK_PAGE_SHIFT;
+
+ if ( 0 == nr_pages )
+ {
+ err = -EINVAL;
+ goto out;
+ }
+
+ kbase_gpu_vm_lock(kctx);
+
+ if (vma->vm_pgoff == KBASE_REG_COOKIE_RB)
+ {
+ /* Reserve offset 0 for the shared ring-buffer */
+ if ((err = kbase_rb_mmap(kctx, vma, ®, &kaddr)))
+ goto out_unlock;
+
+ pr_debug("kbase_rb_mmap ok\n");
+ goto map;
+ }
+ else if (vma->vm_pgoff == KBASE_REG_COOKIE_TB)
+ {
+ err = kbase_trace_buffer_mmap(kctx, vma, ®, &kaddr);
+ if (0 != err)
+ goto out_unlock;
+ pr_debug("kbase_trace_buffer_mmap ok\n");
+ goto map;
+ }
+ else if (vma->vm_pgoff == KBASE_REG_COOKIE_MMU_DUMP)
+ {
+ /* MMU dump */
+ if ((err = kbase_mmu_dump_mmap(kctx, vma, ®, &kaddr)))
+ goto out_unlock;
+
+ goto map;
+ }
+
+ if (vma->vm_pgoff < OSK_PAGE_SIZE) /* first page is reserved for cookie resolution */
+ {
+ /* PMEM stuff, fetch the right region */
+ reg = kbase_lookup_cookie(kctx, vma->vm_pgoff);
+
+ if (NULL != reg)
+ {
+ if (reg->nr_pages != nr_pages)
+ {
+ /* incorrect mmap size */
+ /* leave the cookie for a potential later mapping, or to be reclaimed later when the context is freed */
+ err = -ENOMEM;
+ goto out_unlock;
+ }
+
+ kbase_unlink_cookie(kctx, vma->vm_pgoff, reg);
+
+ if (MALI_ERROR_NONE != kbase_gpu_mmap(kctx, reg, vma->vm_start, nr_pages, 1))
+ {
+ /* Unable to map in GPU space. Recover from kbase_unlink_cookie */
+ OSK_DLIST_PUSH_FRONT(&kctx->osctx.reg_pending, reg, struct kbase_va_region, link);
+ kctx->osctx.cookies &= ~(1UL << vma->vm_pgoff);
+ WARN_ON(1);
+ err = -ENOMEM;
+ goto out_unlock;
+ }
+
+ /*
+ * Overwrite the offset with the
+ * region start_pfn, so we effectively
+ * map from offset 0 in the region.
+ */
+ vma->vm_pgoff = reg->start_pfn;
+ goto map;
+ }
+
+ err = -ENOMEM;
+ goto out_unlock;
+ }
+ else if (vma->vm_pgoff < KBASE_REG_ZONE_EXEC_BASE)
+ {
+ /* invalid offset as it identifies an already mapped pmem */
+ err = -ENOMEM;
+ goto out_unlock;
+ }
+ else
+ {
+ u32 zone;
+
+ /* TMEM case or EXEC case */
+ if (vma->vm_pgoff < KBASE_REG_ZONE_TMEM_BASE)
+ {
+ zone = KBASE_REG_ZONE_EXEC;
+ }
+ else
+ {
+ zone = KBASE_REG_ZONE_TMEM;
+ }
+
+ OSK_DLIST_FOREACH(&kctx->reg_list,
+ struct kbase_va_region, link, reg)
+ {
+ if (reg->start_pfn <= vma->vm_pgoff &&
+ (reg->start_pfn + reg->nr_alloc_pages) >= (vma->vm_pgoff + nr_pages) &&
+ (reg->flags & (KBASE_REG_ZONE_MASK | KBASE_REG_FREE | KBASE_REG_NO_CPU_MAP)) == zone)
+ {
+ /* Match! */
+ goto map;
+ }
+
+ }
+
+ err = -ENOMEM;
+ goto out_unlock;
+ }
+map:
+ err = kbase_cpu_mmap(reg, vma, kaddr, nr_pages);
+
+ if (vma->vm_pgoff == KBASE_REG_COOKIE_MMU_DUMP) {
+ /* MMU dump - userspace should now have a reference on
+ * the pages, so we can now free the kernel mapping */
+ osk_vfree(kaddr);
+ }
+out_unlock:
+ kbase_gpu_vm_unlock(kctx);
+out:
+ if (err)
+ {
+ pr_err("mmap failed %d\n", err);
+ }
+ return err;
+}
+KBASE_EXPORT_TEST_API(kbase_mmap)
+
+mali_error kbase_create_os_context(kbase_os_context *osctx)
+{
+ OSK_ASSERT(osctx != NULL);
+
+ OSK_DLIST_INIT(&osctx->reg_pending);
+ osctx->cookies = ~KBASE_REG_RESERVED_COOKIES;
+ init_waitqueue_head(&osctx->event_queue);
+
+ return MALI_ERROR_NONE;
+}
+KBASE_EXPORT_TEST_API(kbase_create_os_context)
+
+static void kbase_reg_pending_dtor(struct kbase_va_region *reg)
+{
+ kbase_free_phy_pages(reg);
+ pr_info("Freeing pending unmapped region\n");
+ osk_free(reg);
+}
+
+void kbase_destroy_os_context(kbase_os_context *osctx)
+{
+ OSK_ASSERT(osctx != NULL);
+
+ OSK_DLIST_EMPTY_LIST(&osctx->reg_pending, struct kbase_va_region,
+ link, kbase_reg_pending_dtor);
+}
+KBASE_EXPORT_TEST_API(kbase_destroy_os_context)
+
+void *kbase_va_alloc(kbase_context *kctx, u32 size)
+{
+ void *va;
+ u32 pages = ((size-1) >> OSK_PAGE_SHIFT) + 1;
+ struct kbase_va_region *reg;
+ osk_phy_addr *page_array;
+ u32 flags = BASE_MEM_PROT_CPU_RD | BASE_MEM_PROT_CPU_WR |
+ BASE_MEM_PROT_GPU_RD | BASE_MEM_PROT_GPU_WR;
+ int i;
+
+ OSK_ASSERT(kctx != NULL);
+
+ if (size == 0)
+ {
+ goto err;
+ }
+
+ va = osk_vmalloc(size);
+ if (!va)
+ {
+ goto err;
+ }
+
+ kbase_gpu_vm_lock(kctx);
+
+ reg = kbase_alloc_free_region(kctx, 0, pages, KBASE_REG_ZONE_PMEM);
+ if (!reg)
+ {
+ goto vm_unlock;
+ }
+
+ reg->flags &= ~KBASE_REG_FREE;
+ kbase_update_region_flags(reg, flags, MALI_FALSE);
+
+ reg->nr_alloc_pages = pages;
+ reg->extent = 0;
+
+ page_array = osk_vmalloc(pages * sizeof(*page_array));
+ if (!page_array)
+ {
+ goto free_reg;
+ }
+
+ for (i = 0; i < pages; i++)
+ {
+ uintptr_t addr;
+ struct page *page;
+ addr = (uintptr_t)va + (i << OSK_PAGE_SHIFT);
+ page = vmalloc_to_page((void *)addr);
+ page_array[i] = PFN_PHYS(page_to_pfn(page));
+ }
+
+ kbase_set_phy_pages(reg, page_array);
+
+ if (kbase_gpu_mmap(kctx, reg, (uintptr_t)va, pages, 1))
+ {
+ goto free_array;
+ }
+
+ kbase_gpu_vm_unlock(kctx);
+
+ return va;
+
+free_array:
+ osk_vfree(page_array);
+free_reg:
+ osk_free(reg);
+vm_unlock:
+ kbase_gpu_vm_lock(kctx);
+ osk_vfree(va);
+err:
+ return NULL;
+}
+KBASE_EXPORT_SYMBOL(kbase_va_alloc)
+
+void kbase_va_free(kbase_context *kctx, void *va)
+{
+ struct kbase_va_region *reg;
+ osk_phy_addr *page_array;
+ mali_error err;
+
+ OSK_ASSERT(kctx != NULL);
+ OSK_ASSERT(va != NULL);
+
+ kbase_gpu_vm_lock(kctx);
+
+ reg = kbase_validate_region(kctx, (uintptr_t)va);
+ OSK_ASSERT(reg);
+
+ err = kbase_gpu_munmap(kctx, reg);
+ OSK_ASSERT(err == MALI_ERROR_NONE);
+
+ page_array = kbase_get_phy_pages(reg);
+ osk_vfree(page_array);
+
+ osk_free(reg);
+
+ kbase_gpu_vm_unlock(kctx);
+
+ osk_vfree(va);
+}
+KBASE_EXPORT_SYMBOL(kbase_va_free)
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010, 2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_kbase_mem_linux.h
+ * Base kernel memory APIs, Linux implementation.
+ */
+
+#ifndef _KBASE_MEM_LINUX_H_
+#define _KBASE_MEM_LINUX_H_
+
+struct kbase_va_region *kbase_pmem_alloc(struct kbase_context *kctx, u32 size,
+ u32 flags, u16 *pmem_cookie);
+int kbase_mmap(struct file *file, struct vm_area_struct *vma);
+
+/* @brief Allocate memory from kernel space and map it onto the GPU
+ *
+ * @param kctx The context used for the allocation/mapping
+ * @param size The size of the allocation in bytes
+ * @return the VA for kernel space and GPU MMU
+ */
+void *kbase_va_alloc(kbase_context *kctx, u32 size);
+
+/* @brief Free/unmap memory allocated by kbase_va_alloc
+ *
+ * @param kctx The context used for the allocation/mapping
+ * @param va The VA returned by kbase_va_alloc
+ */
+void kbase_va_free(kbase_context *kctx, void *va);
+
+#endif /* _KBASE_MEM_LINUX_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#if !defined(_TRACE_MALI_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_MALI_H
+
+#include <linux/stringify.h>
+#include <linux/tracepoint.h>
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM mali
+#define TRACE_SYSTEM_STRING __stringify(TRACE_SYSTEM)
+#define TRACE_INCLUDE_FILE mali_linux_trace
+
+/**
+ * mali_job_slots_event - called from mali_kbase_core_linux.c
+ * @event_id: ORed together bitfields representing a type of event, made with the GATOR_MAKE_EVENT() macro.
+ */
+TRACE_EVENT(mali_job_slots_event,
+
+ TP_PROTO(unsigned int event_id),
+
+ TP_ARGS(event_id),
+
+ TP_STRUCT__entry(
+ __field( int, event_id )
+ ),
+
+ TP_fast_assign(
+ __entry->event_id = event_id;
+ ),
+
+ TP_printk("event=%d", __entry->event_id)
+);
+
+/**
+ * mali_timeline_event - not currently used
+ * @event_id: ORed together bitfields representing a type of event, made with the GATOR_MAKE_EVENT() macro.
+ */
+TRACE_EVENT(mali_timeline_event,
+
+ TP_PROTO(unsigned int event_id),
+
+ TP_ARGS(event_id),
+
+ TP_STRUCT__entry(
+ __field( int, event_id )
+ ),
+
+ TP_fast_assign(
+ __entry->event_id = event_id;
+ ),
+
+ TP_printk("event=%d", __entry->event_id)
+);
+
+/**
+ * mali_hw_counter - not currently used
+ */
+TRACE_EVENT(mali_hw_counter,
+
+ TP_PROTO(unsigned int event_id, unsigned int value),
+
+ TP_ARGS(event_id, value),
+
+ TP_STRUCT__entry(
+ __field( int, event_id )
+ __field( int, value )
+ ),
+
+ TP_fast_assign(
+ __entry->event_id = event_id;
+ ),
+
+ TP_printk("event %d = %d", __entry->event_id, __entry->value)
+);
+
+/**
+ * mali_pm_status - Called by mali_kbase_pm_driver.c
+ * @event_id: core type (shader, tiler, l2 cache, l3 cache)
+ * @value: 64bits bitmask reporting either power status of the cores (1-ON, 0-OFF)
+ */
+TRACE_EVENT(mali_pm_status,
+
+ TP_PROTO(unsigned int event_id, unsigned long long value),
+
+ TP_ARGS(event_id, value),
+
+ TP_STRUCT__entry(
+ __field( unsigned int, event_id )
+ __field( unsigned long long, value )
+ ),
+
+ TP_fast_assign(
+ __entry->event_id = event_id;
+ ),
+
+ TP_printk("event %u = %llu", __entry->event_id, __entry->value)
+);
+
+/**
+ * mali_pm_power_on - Called by mali_kbase_pm_driver.c
+ * @event_id: core type (shader, tiler, l2 cache, l3 cache)
+ * @value: 64bits bitmask reporting the cores to power up
+ */
+TRACE_EVENT(mali_pm_power_on,
+
+ TP_PROTO(unsigned int event_id, unsigned long long value),
+
+ TP_ARGS(event_id, value),
+
+ TP_STRUCT__entry(
+ __field( unsigned int, event_id )
+ __field( unsigned long long, value )
+ ),
+
+ TP_fast_assign(
+ __entry->event_id = event_id;
+ ),
+
+ TP_printk("event %u = %llu", __entry->event_id, __entry->value)
+);
+
+/**
+ * mali_pm_power_off - Called by mali_kbase_pm_driver.c
+ * @event_id: core type (shader, tiler, l2 cache, l3 cache)
+ * @value: 64bits bitmask reporting the cores to power down
+ */
+TRACE_EVENT(mali_pm_power_off,
+
+ TP_PROTO(unsigned int event_id, unsigned long long value),
+
+ TP_ARGS(event_id, value),
+
+ TP_STRUCT__entry(
+ __field( unsigned int, event_id )
+ __field( unsigned long long, value )
+ ),
+
+ TP_fast_assign(
+ __entry->event_id = event_id;
+ ),
+
+ TP_printk("event %u = %llu", __entry->event_id, __entry->value)
+);
+
+/**
+ * mali_page_fault_insert_pages - Called by page_fault_worker()
+ * it reports an MMU page fault resulting in new pages being mapped.
+ * @event_id: MMU address space number.
+ * @value: number of newly allocated pages
+ */
+TRACE_EVENT(mali_page_fault_insert_pages,
+
+ TP_PROTO(int event_id, unsigned long value),
+
+ TP_ARGS(event_id, value),
+
+ TP_STRUCT__entry(
+ __field( int, event_id )
+ __field( unsigned long, value )
+ ),
+
+ TP_fast_assign(
+ __entry->event_id = event_id;
+ ),
+
+ TP_printk("event %d = %lu", __entry->event_id, __entry->value)
+);
+
+/**
+ * mali_mmu_as_in_use - Called by assign_and_activate_kctx_addr_space()
+ * it reports that a certain MMU address space is in use now.
+ * @event_id: MMU address space number.
+ */
+TRACE_EVENT(mali_mmu_as_in_use,
+
+ TP_PROTO(int event_id),
+
+ TP_ARGS(event_id),
+
+ TP_STRUCT__entry(
+ __field( int, event_id )
+ ),
+
+ TP_fast_assign(
+ __entry->event_id = event_id;
+ ),
+
+ TP_printk("event=%d", __entry->event_id)
+);
+
+/**
+ * mali_mmu_as_released - Called by kbasep_js_runpool_release_ctx_internal()
+ * it reports that a certain MMU address space has been released now.
+ * @event_id: MMU address space number.
+ */
+TRACE_EVENT(mali_mmu_as_released,
+
+ TP_PROTO(int event_id),
+
+ TP_ARGS(event_id),
+
+ TP_STRUCT__entry(
+ __field( int, event_id )
+ ),
+
+ TP_fast_assign(
+ __entry->event_id = event_id;
+ ),
+
+ TP_printk("event=%d", __entry->event_id)
+);
+
+/**
+ * mali_total_alloc_pages_change - Called by kbase_mem_usage_request_pages()
+ * and by kbase_mem_usage_release_pages
+ * it reports that the total number of allocated pages is changed.
+ * @event_id: number of pages to be added or subtracted (according to the sign).
+ */
+TRACE_EVENT(mali_total_alloc_pages_change,
+
+ TP_PROTO(long long int event_id),
+
+ TP_ARGS(event_id),
+
+ TP_STRUCT__entry(
+ __field( long long int, event_id )
+ ),
+
+ TP_fast_assign(
+ __entry->event_id = event_id;
+ ),
+
+ TP_printk("event=%lld", __entry->event_id)
+);
+
+/**
+ * mali_sw_counter - not currently used
+ * @event_id: counter id
+ */
+TRACE_EVENT(mali_sw_counter,
+
+ TP_PROTO(unsigned int event_id, signed long long value),
+
+ TP_ARGS(event_id, value),
+
+ TP_STRUCT__entry(
+ __field( int, event_id )
+ __field( long long, value )
+ ),
+
+ TP_fast_assign(
+ __entry->event_id = event_id;
+ ),
+
+ TP_printk("event %d = %lld", __entry->event_id, __entry->value)
+);
+
+#endif /* _TRACE_MALI_H */
+
+#undef TRACE_INCLUDE_PATH
+#undef linux
+#define TRACE_INCLUDE_PATH MALI_KBASE_SRC_LINUX_PATH
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _BASE_MEM_PRIV_H_
+#define _BASE_MEM_PRIV_H_
+
+#define BASE_SYNCSET_OP_MSYNC (1U << 0)
+#define BASE_SYNCSET_OP_CSYNC (1U << 1)
+
+/*
+ * This structure describe a basic memory coherency operation.
+ * It can either be:
+ * @li a sync from CPU to Memory:
+ * - type = ::BASE_SYNCSET_OP_MSYNC
+ * - mem_handle = a handle to the memory object on which the operation
+ * is taking place
+ * - user_addr = the address of the range to be synced
+ * - size = the amount of data to be synced, in bytes
+ * - offset is ignored.
+ * @li a sync from Memory to CPU:
+ * - type = ::BASE_SYNCSET_OP_CSYNC
+ * - mem_handle = a handle to the memory object on which the operation
+ * is taking place
+ * - user_addr = the address of the range to be synced
+ * - size = the amount of data to be synced, in bytes.
+ * - offset is ignored.
+ */
+typedef struct basep_syncset
+{
+ mali_addr64 mem_handle;
+ u64 user_addr;
+ u32 size;
+ u8 type;
+} basep_syncset;
+
+#endif
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010, 2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+#ifndef _BASE_VENDOR_SPEC_FUNC_H_
+#define _BASE_VENDOR_SPEC_FUNC_H_
+
+#include <malisw/mali_stdtypes.h>
+
+mali_error kbase_get_vendor_specific_cpu_clock_speed(u32*);
+
+#endif /*_BASE_VENDOR_SPEC_FUNC_H_*/
--- /dev/null
+# Copyright:
+# ----------------------------------------------------------------------------
+# This confidential and proprietary software may be used only as authorized
+# by a licensing agreement from ARM Limited.
+# (C) COPYRIGHT 2010-2012 ARM Limited, ALL RIGHTS RESERVED
+# The entire notice above must be reproduced on all authorized copies and
+# copies may only be made to the extent permitted by a licensing agreement
+# from ARM Limited.
+# ----------------------------------------------------------------------------
+#
+
+import os
+import re
+import sys
+Import('env')
+
+scheduling_policy = 'cfs'
+mock_test = 0
+
+if env['error_inject'] == '1':
+ env.Append( CPPDEFINES = {'MALI_ERROR_INJECT_ON' : 1} )
+elif env['error_inject'] == '2':
+ env.Append( CPPDEFINES = {'MALI_ERROR_INJECT_ON' : 2} )
+else:
+ env['error_inject'] = 0
+ env.Append( CPPDEFINES = {'MALI_ERROR_INJECT_ON' : 0} )
+
+if env['hwver'] == 'none':
+ env['mali_kbasep_model']='1'
+else:
+ env['mali_kbasep_model']='0'
+env.Append( CPPDEFINES = { 'MALI_KBASEP_MODEL': env['mali_kbasep_model'] } )
+
+if env['os'] == 'linux' or env['os'] == 'android':
+ if env['backend'] == 'kernel':
+ if env['v'] != '1':
+ env['MAKECOMSTR'] = '[MAKE] ${SOURCE.dir}'
+
+ # Fake platform is a transient solution for GPL drivers running in kernel that does not provide configuration via platform data.
+ # For such kernels fake_platform_device should be set to 1. For kernels providing platform data and for
+ # commercial driver fake_platform_device should be set to 0.
+ if int(env['mali_license_is_gpl']) == 1:
+ fake_platform_device = 1
+ else:
+ fake_platform_device = 0
+
+ # Source files required for kbase.
+ kbase_src = [Glob('#kbase/src/common/*.c'), Glob('#kbase/src/linux/*.c'), Glob('#kbase/src/common/*.h'), Glob('#kbase/src/linux/*.h')]
+
+ if Glob('#kbase/tests/internal/src/mock') and env['unit'] == '1':
+ kbase_src += [Glob('#kbase/tests/internal/src/mock/*.c')]
+ mock_test = 1
+
+ # we need platform config for commercial version of the driver and for GPL version using fake platform
+ if int(env['mali_license_is_gpl']) == 0 or fake_platform_device==1:
+ # Check if we are compiling for PBX
+ linux_config_file = os.path.normpath(os.environ['KDIR']) + '/.config'
+ search_term = '^[\ ]*CONFIG_MACH_REALVIEW_PBX[\ ]*=[\ ]*y'
+ REALVIEW_PBX = 0
+ for line in open(linux_config_file, 'r'):
+ if re.search(search_term, line):
+ REALVIEW_PBX = 1
+ break
+ if REALVIEW_PBX == 1 and env['platform_config'] == 'vexpress':
+ sys.stderr.write("WARNING: Building for a PBX kernel but with platform_config=vexpress\n")
+ # if the file platform config file is in the tpip directory then use that, otherwise use the default config directory
+ if Glob('#kbase/src/linux/config/tpip/*%s.c' % (env['platform_config'])):
+ kbase_src += Glob('#kbase/src/linux/config/tpip/*%s.c' % (env['platform_config']))
+ else:
+ kbase_src += Glob('#kbase/src/linux/config/*%s.c' % (env['platform_config']))
+
+ # Note: cleaning via the Linux kernel build system does not yet work
+ if env.GetOption('clean') :
+ makeAction=Action("cd ${SOURCE.dir}/.. && make clean", '$MAKECOMSTR')
+ else:
+ if env['os'] == 'android':
+ env['android'] = 1
+ else:
+ env['android'] = 0
+
+ if env['unit'] == '1':
+ env['kernel_test'] = 1
+ else:
+ env['kernel_test'] = 0
+ makeAction=Action("cd ${SOURCE.dir}/.. && make PLATFORM=${platform} MALI_KBASEP_MODEL=${mali_kbasep_model} MALI_ERROR_INJECT_ON=${error_inject} MALI_BACKEND_KERNEL=1 MALI_NO_MALI=${no_mali} MALI_USE_UMP=${ump} MALI_DEBUG=${debug} MALI_ANDROID=${android} MALI_BASE_TRACK_MEMLEAK=${base_qa} MALI_KERNEL_TEST_API=${kernel_test} MALI_KBASE_SCHEDULING_POLICY=%s MALI_UNIT_TEST=${unit} MALI_INFINITE_CACHE=${infinite_cache} MALI_LICENSE_IS_GPL=${mali_license_is_gpl} MALI_PLATFORM_CONFIG=${platform_config} MALI_ERROR_INJECT_ON=${error_inject} MALI_UNCACHED=${no_syncsets} MALI_RELEASE_NAME=\"${mali_release_name}\" MALI_FAKE_PLATFORM_DEVICE=%s MALI_MOCK_TEST=%s MALI_GATOR_SUPPORT=${gator} MALI_CUSTOMER_RELEASE=${release} MALI_INSTRUMENTATION_LEVEL=${instr} MALI_COVERAGE=${coverage} && cp mali_kbase.ko $STATIC_LIB_PATH/mali_kbase.ko" % (scheduling_policy, fake_platform_device, mock_test), '$MAKECOMSTR')
+
+ cmd = env.Command('$STATIC_LIB_PATH/mali_kbase.ko', kbase_src, [makeAction])
+
+ if int(env['mali_license_is_gpl']) == 1:
+ env.Depends('$STATIC_LIB_PATH/mali_kbase.ko', '$STATIC_LIB_PATH/kds.ko')
+
+ env.Depends('$STATIC_LIB_PATH/mali_kbase.ko', '$STATIC_LIB_PATH/libosk.a')
+ # need Module.symvers from ukk.kko and ump.ko builds
+ env.Depends('$STATIC_LIB_PATH/mali_kbase.ko', '$STATIC_LIB_PATH/ukk.ko')
+ if int(env['ump']) == 1:
+ env.Depends('$STATIC_LIB_PATH/mali_kbase.ko', '$STATIC_LIB_PATH/ump.ko')
+
+ # Until we fathom out how the invoke the Linux build system to clean, we can use Clean
+ # to remove generated files.
+ patterns = ['*.mod.c', '*.o', '*.ko', '*.a', '.*.cmd', 'modules.order', '.tmp_versions', 'Module.symvers']
+
+ for p in patterns:
+ Clean(cmd, Glob('#kbase/src/%s' % p))
+ Clean(cmd, Glob('#kbase/src/linux/%s' % p))
+ Clean(cmd, Glob('#kbase/src/linux/config/%s' % p))
+ Clean(cmd, Glob('#kbase/src/linux/config/tpip/%s' % p))
+ Clean(cmd, Glob('#kbase/src/common/%s' % p))
+ Clean(cmd, Glob('#kbase/tests/internal/src/mock/%s' % p))
+
+ env.ProgTarget('kbase', cmd)
+
+ env.AppendUnique(BASE=['cutils_list'])
+ else:
+ common_source = [
+ 'common/mali_kbase_cache_policy.c',
+ 'common/mali_kbase_mem.c',
+ 'common/mali_kbase_mmu.c',
+ 'common/mali_kbase_jd.c',
+ 'common/mali_kbase_jm.c',
+ 'common/mali_kbase_js.c',
+ 'common/mali_kbase_js_affinity.c',
+ 'common/mali_kbase_js_ctx_attr.c',
+ 'common/mali_kbase_js_policy_%s.c' % (scheduling_policy),
+ 'common/mali_kbase_pm.c',
+ 'common/mali_kbase_cpuprops.c',
+ 'common/mali_kbase_gpuprops.c',
+ 'common/mali_kbase_event.c',
+ 'common/mali_kbase_context.c',
+ 'common/mali_kbase_pm.c',
+ 'common/mali_kbase_pm_driver.c',
+ 'common/mali_kbase_pm_metrics.c',
+ 'common/mali_kbase_pm_always_on.c',
+ 'common/mali_kbase_pm_demand.c',
+ 'common/mali_kbase_device.c',
+ 'common/mali_kbase_config.c',
+ 'common/mali_kbase_security.c',
+ 'common/mali_kbase_instr.c',
+ 'common/mali_kbase_8401_workaround.c',
+ 'common/mali_kbase_softjobs.c',
+ 'common/mali_kbase_hw.c',
+ 'userspace/mali_kbase_core_userspace.c',
+ 'userspace/mali_kbase_model_userspace.c',
+ 'userspace/mali_kbase_mem_userspace.c',
+ 'userspace/mali_kbase_ump.c',
+ 'userspace/mali_kbase_pm_metrics_userspace.c'
+ ]
+
+ if Glob('#kbase/tests/internal/src/mock') and env['unit'] == '1':
+ common_source += ['../tests/internal/src/mock/mali_kbase_pm_driver_mock.c']
+ mock_test = 1
+
+ os_source = []
+
+ if env['os'] in ['linux']:
+ pass
+ else:
+ sys.stderr.write("*** Unsupported OS: %s\n" % env['os'])
+ Exit(1)
+
+ env.Append( CPPDEFINES = {'MALI_KBASE_USERSPACE' : 1} )
+ env.Append( CPPDEFINES = {'MALI_MOCK_TEST' : mock_test} )
+
+ if env['backend'] == 'user' and env['no_mali'] == '1':
+ hwsim_source = ['common/mali_kbase_model_dummy.c',
+ 'common/mali_kbase_model_error_generator.c']
+ env.AppendUnique(BASE=['cutils_list', 'kbase'])
+ else:
+ # Unpack and extract the model - will only work on x86 Linux
+ if env['arch'] == 'x86_64':
+ hostbits = '64'
+ else:
+ hostbits = '32'
+
+ # Create a builder to handle extracting the model binary from the tarball. Using a builder,
+ # we can define a custom COMSTR to give less verbose output if requested.
+ extract = Action('tar xzf $SOURCE --strip-components 4 Rexported/lib/x86_rhe5_%s/%s/libMidgardModel.so -O > $TARGET' % (hostbits, env['model']), "$EXTRACTCOMSTR")
+ extract_builder = Builder(action = extract)
+ env.Append(BUILDERS = {'Extract' : extract_builder})
+ if not int(env['v']):
+ env.Append(EXTRACTCOMSTR = '[EXTRACT] $TARGET')
+
+ # Any builds dependent on the target "model" will cause the binary to be extracted
+ # Note that to maintain compatability with existing build files that expect to link against
+ # the static version, we extract to $STATIC_LIB_PATH too.
+ model = env.Extract('$STATIC_LIB_PATH/libMidgardModel.so','#model/model.tgz')
+ model_dlib = env.Extract('$SHARED_LIB_PATH/libMidgardModel.so','#model/model.tgz')
+ env.Depends(model, model_dlib)
+
+ # Create an action that can be used as a post-action, to install the model whenever it is unpacked,
+ # provding that the command-line option "libs_install" has been set. We also add a Clean method
+ # to delete the installed model when the extracted model is cleaned.
+ if env.has_key('libs_install'):
+ a = Action("mkdir -p {libs}; cp $STATIC_LIB_PATH/libMidgardModel.so {libs}".format(libs=env['libs_install']), "$COPYCOMSTR")
+ if not int(env['v']):
+ env.Append(COPYCOMSTR = '[COPY] $TARGET')
+ env.AddPostAction(model, a)
+ env.Clean(model, os.path.join(env['libs_install'], "libMidgardModel.so"))
+
+
+ hwsim_source = []
+ env.AppendUnique(
+ BASE=['cutils_list', 'kbase', 'MidgardModel', 'stdc++']
+ )
+ env.Alias('kbase', model)
+
+ cppdefines = dict(env['CPPDEFINES'])
+ if env['unit'] == '1':
+ #make a local definition for STATIC
+ cppdefines.update( {'STATIC':''} )
+
+ libs=env.StaticLibrary( '$STATIC_LIB_PATH/kbase', [common_source, os_source, hwsim_source], CPPDEFINES = cppdefines)
+ env.LibTarget('kbase', libs)
--- /dev/null
+obj-y += src/
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_atomics.h
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ATOMICS_H_
+#define _OSK_ATOMICS_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @defgroup oskatomic Atomic Access
+ *
+ * @anchor oskatomic_important
+ * @par Important Information on Atomic variables
+ *
+ * Atomic variables are objects that can be modified by only one thread at a time.
+ * For use in SMP systems, strongly ordered access is enforced using memory
+ * barriers.
+ *
+ * An atomic variable implements an unsigned integer counter which is exactly
+ * 32 bits long. Arithmetic on it is the same as on u32 values, which is the
+ * arithmetic of integers modulo 2^32. For example, incrementing past
+ * 0xFFFFFFFF rolls over to 0, decrementing past 0 rolls over to
+ * 0xFFFFFFFF. That is, overflow is a well defined condition (unlike signed
+ * integer arithmetic in C).
+ */
+/** @{ */
+
+/** @brief Subtract a value from an atomic variable and return the new value.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a atom parameter.
+ *
+ * @note Please refer to @see oskatomic_important Important Information on Atomic
+ * variables.
+ *
+ * @param atom pointer to an atomic variable
+ * @param value value to subtract from \a atom
+ * @return value of atomic variable after \a value has been subtracted from it.
+ */
+OSK_STATIC_INLINE u32 osk_atomic_sub(osk_atomic *atom, u32 value);
+
+/** @brief Add a value to an atomic variable and return the new value.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a atom parameter.
+ *
+ * @note Please refer to @see oskatomic_important Important Information on Atomic
+ * variables.
+ *
+ * @param atom pointer to an atomic variable
+ * @param value value to add to \a atom
+ * @return value of atomic variable after \a value has been added to it.
+ */
+OSK_STATIC_INLINE u32 osk_atomic_add(osk_atomic *atom, u32 value);
+
+/** @brief Decrement an atomic variable and return its decremented value.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a atom parameter.
+ *
+ * @note Please refer to @see oskatomic_important Important Information on Atomic
+ * variables.
+ *
+ * @param atom pointer to an atomic variable
+ * @return decremented value of atomic variable
+ */
+OSK_STATIC_INLINE u32 osk_atomic_dec(osk_atomic *atom);
+
+/** @brief Increment an atomic variable and return its incremented value.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a atom parameter.
+ *
+ * @note Please refer to @see oskatomic_important Important Information on Atomic
+ * variables.
+ *
+ * @param atom pointer to an atomic variable
+ * @return incremented value of atomic variable
+ */
+OSK_STATIC_INLINE u32 osk_atomic_inc(osk_atomic *atom);
+
+/** @brief Sets the value of an atomic variable.
+ *
+ * Note: if the value of the atomic variable is set as part of a read-modify-write
+ * operation and multiple threads have access to the atomic variable at that time,
+ * please use osk_atomic_compare_and_swap() instead, which can ensure no other
+ * process changed the atomic variable during the read-write-modify operation.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a atom parameter.
+ *
+ * @note Please refer to @see oskatomic_important Important Information on Atomic
+ * variables.
+ *
+ * @param atom pointer to an atomic variable
+ * @param value the value to set
+ */
+OSK_STATIC_INLINE void osk_atomic_set(osk_atomic *atom, u32 value);
+
+/** @brief Return the value of an atomic variable.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a atom parameter.
+ *
+ * @note Please refer to @see oskatomic_important Important Information on Atomic
+ * variables.
+ *
+ * @param atom pointer to an atomic variable
+ * @return value of the atomic variable
+ */
+OSK_STATIC_INLINE u32 osk_atomic_get(osk_atomic *atom);
+
+/** @brief Compare the value of an atomic variable, and atomically exchange it
+ * if the comparison succeeds.
+ *
+ * This function implements the Atomic Compare-And-Swap operation (CAS) which
+ * allows atomically performing a read-modify-write operation on atomic variables.
+ * The CAS operation is suited for implementing synchronization primitives such
+ * as semaphores and mutexes, as well as lock-free and wait-free algorithms.
+ *
+ * It atomically does the following: compare \a atom with \a old_value and sets \a
+ * atom to \a new_value if the comparison was true.
+ *
+ * Regardless of the outcome of the comparison, the initial value of \a atom is
+ * returned - hence the reason for this being a 'swap' operation. If the value
+ * returned is equal to \a old_value, then the atomic operation succeeded. Any
+ * other value shows that the atomic operation failed, and should be repeated
+ * based upon the returned value.
+ *
+ * For example:
+@code
+typedef struct my_data
+{
+ osk_atomic index;
+ object objects[10];
+} data;
+u32 index, old_index, new_index;
+
+// Updates the index into an array of objects based on the current indexed object.
+// If another process updated the index in the mean time, the index will not be
+// updated and we try again based on the updated index.
+
+index = osk_atomic_get(&data.index);
+do {
+ old_index = index;
+ new_index = calc_new_index(&data.objects[old_index]);
+ index = osk_atomic_compare_and_swap(&data.index, old_index, new_index)
+} while (index != old_index);
+
+@endcode
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a atom parameter.
+ *
+ * @note Please refer to @see oskatomic_important Important Information on Atomic
+ * variables.
+ *
+ * @param atom pointer to an atomic variable
+ * @param old_value The value to make the comparison with \a atom
+ * @param new_value The value to atomically write to atom, depending on whether
+ * the comparison succeeded.
+ * @return The \em initial value of \a atom, before the operation commenced.
+ */
+OSK_STATIC_INLINE u32 osk_atomic_compare_and_swap(osk_atomic * atom, u32 old_value, u32 new_value) CHECK_RESULT;
+
+/** @} */ /* end group oskatomic */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+/* pull in the arch header with the implementation */
+#include <osk/mali_osk_arch_atomics.h>
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _OSK_ATOMICS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_bitops.h
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_BITOPS_H_
+#define _OSK_BITOPS_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+#include <osk/mali_osk_arch_bitops.h>
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/** @defgroup oskbitops Bit-operations
+ *
+ * These bit-operations do not work atomically, and so locks must be used if
+ * atomicity is required.
+ *
+ * Reference implementations for Little Endian are provided, and so it should
+ * not normally be necessary to re-implement these. Efficient bit-twiddling
+ * techniques are used where possible, implemented in portable C.
+ *
+ * Note that these reference implementations rely on osk_clz() being
+ * implemented.
+ *
+ * @{
+ */
+
+/**
+ * @brief Tests if a bit is set in a unsigned long value (internal function)
+ * @param[in] bit bit number to test [0..OSK_BITS_PER_LONG-1], starting from the (Little-endian) least significant bit
+ * @param[in] value unsigned long value
+ * @return zero if bit was clear, non-zero if set. Do not rely on the return
+ * value being related to the actual word under test.
+ */
+OSK_STATIC_INLINE unsigned long oskp_test_bit(unsigned long bit, unsigned long value)
+{
+ OSK_ASSERT( bit < OSK_BITS_PER_LONG );
+
+ return value & (1UL << bit);
+}
+
+/**
+ * @brief Find the first zero bit in a unsigned long value
+ * @param[in] value unsigned long value
+ * @return a postive number [0..OSK_BITS_PER_LONG-1], starting from the least significant bit,
+ * indicating the first zero bit found, or a negative number if no zero bit was found.
+ */
+CHECK_RESULT OSK_STATIC_INLINE long oskp_find_first_zero_bit(unsigned long value)
+{
+ unsigned long inverted;
+ unsigned long negated;
+ unsigned long isolated;
+ unsigned long leading_zeros;
+
+ /* Begin with xxx...x0yyy...y, where ys are 1, number of ys is in range 0..31/63 */
+ inverted = ~value; /* zzz...z1000...0 */
+ /* Using count_trailing_zeros on inverted value -
+ * See ARM System Developers Guide for details of count_trailing_zeros */
+
+ /* Isolate the zero: it is preceeded by a run of 1s, so add 1 to it */
+ negated = (unsigned long)-inverted ; /* -a == ~a + 1 (mod 2^n) for n-bit numbers */
+ /* negated = xxx...x1000...0 */
+
+ isolated = negated & inverted ; /* xxx...x1000...0 & zzz...z1000...0, zs are ~xs */
+ /* And so the first zero bit is in the same position as the 1 == number of 1s that preceeded it
+ * Note that the output is zero if value was all 1s */
+
+ leading_zeros = osk_clz( isolated );
+
+ return (OSK_BITS_PER_LONG - 1) - leading_zeros;
+}
+
+/**
+ * @brief Clear a bit in a sequence of unsigned longs
+ * @param[in] nr bit number to clear, starting from the (Little-endian) least
+ * significant bit
+ * @param[in,out] addr starting point for counting.
+ */
+OSK_STATIC_INLINE void osk_bitarray_clear_bit(unsigned long nr, unsigned long *addr )
+{
+ OSK_ASSERT(NULL != addr);
+ addr += nr / OSK_BITS_PER_LONG; /* find the correct word */
+ nr = nr & (OSK_BITS_PER_LONG - 1); /* The bit number within the word */
+ *addr &= ~(1UL << nr);
+}
+
+/**
+ * @brief Set a bit in a sequence of unsigned longs
+ * @param[in] nr bit number to set, starting from the (Little-endian) least
+ * significant bit
+ * @param[in,out] addr starting point for counting.
+ */
+OSK_STATIC_INLINE void osk_bitarray_set_bit(unsigned long nr, unsigned long *addr)
+{
+ OSK_ASSERT(NULL != addr);
+ addr += nr / OSK_BITS_PER_LONG; /* find the correct word */
+ nr = nr & (OSK_BITS_PER_LONG - 1); /* The bit number within the word */
+ *addr |= (1UL << nr);
+}
+
+/**
+ * @brief Test a bit in a sequence of unsigned longs
+ * @param[in] nr bit number to test, starting from the (Little-endian) least
+ * significant bit
+ * @param[in,out] addr starting point for counting.
+ * @return zero if bit was clear, non-zero if set. Do not rely on the return
+ * value being related to the actual word under test.
+ */
+CHECK_RESULT OSK_STATIC_INLINE unsigned long osk_bitarray_test_bit(unsigned long nr, unsigned long *addr)
+{
+ OSK_ASSERT(NULL != addr);
+ addr += nr / OSK_BITS_PER_LONG; /* find the correct word */
+ nr = nr & (OSK_BITS_PER_LONG - 1); /* The bit number within the word */
+ return *addr & (1UL << nr);
+}
+
+/**
+ * @brief Find the first zero bit in a sequence of unsigned longs
+ * @param[in] addr starting point for search.
+ * @param[in] maxbit the maximum number of bits to search
+ * @return the number of the first zero bit found, or maxbit if none were found
+ * in the specified range.
+ */
+CHECK_RESULT unsigned long osk_bitarray_find_first_zero_bit(const unsigned long *addr, unsigned long maxbit);
+
+/**
+ * @brief Find the first set bit in a unsigned long
+ * @param val value to find first set bit in
+ * @return the number of the set bit found (starting from 0), -1 if no bits set
+ */
+CHECK_RESULT OSK_STATIC_INLINE long osk_find_first_set_bit(unsigned long val)
+{
+ return (OSK_BITS_PER_LONG - 1) - osk_clz( val & -val );
+}
+
+/**
+ * @brief Count leading zeros in an unsigned long
+ *
+ * Same behavior as ARM CLZ instruction.
+ *
+ * Returns the number of binary zero bits before the first (most significant)
+ * binary one bit in \a val.
+ *
+ * If \a val is zero, this function returns the number of bits in an unsigned
+ * long, ie. sizeof(unsigned long) * 8.
+ *
+ * @param val unsigned long value to count leading zeros in
+ * @return the number of leading zeros
+ */
+CHECK_RESULT OSK_STATIC_INLINE long osk_clz(unsigned long val);
+
+/**
+ * @brief Count leading zeros in an u64
+ *
+ * Same behavior as ARM CLZ instruction.
+ *
+ * Returns the number of binary zero bits before the first (most significant)
+ * binary one bit in \a val.
+ *
+ * If \a val is zero, this function returns the number of bits in an u64,
+ * ie. sizeof(u64) * 8 = 64
+ *
+ * Note that on platforms where an unsigned long is 64 bits then this is the same as osk_clz.
+ *
+ * @param val value to count leading zeros in
+ * @return the number of leading zeros
+ */
+CHECK_RESULT OSK_STATIC_INLINE long osk_clz_64(u64 val);
+
+/**
+ * @brief Count the number of bits set in an unsigned long
+ *
+ * This returns the number of bits set in a unsigned long value.
+ *
+ * @param val The value to count bits set in
+ * @return The number of bits set in \c val.
+ */
+OSK_STATIC_INLINE int osk_count_set_bits(unsigned long val) CHECK_RESULT;
+
+/**
+ * @brief Count the number of bits set in an u64
+ *
+ * This returns the number of bits set in a u64 value.
+ *
+ * @param val The value to count bits set in
+ * @return The number of bits set in \c val.
+ */
+CHECK_RESULT OSK_STATIC_INLINE int osk_count_set_bits64(u64 val)
+{
+ return osk_count_set_bits(val & U32_MAX)
+ + osk_count_set_bits((val >> 32) & U32_MAX);
+}
+
+/** @} */ /* end group oskbitops */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _OSK_BITOPS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_credentials.h
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_CREDENTIALS_H_
+#define _OSK_CREDENTIALS_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @defgroup oskcredentials Credentials Access
+ */
+/** @{ */
+
+/** @brief Check if the caller is privileged.
+ *
+ * @return MALI_TRUE if the caller is privileged.
+ */
+OSK_STATIC_INLINE mali_bool osk_is_privileged(void);
+
+#define OSK_PROCESS_PRIORITY_MIN ( -20 )
+#define OSK_PROCESS_PRIORITY_MAX ( 19 )
+
+typedef struct osk_process_priority
+{
+ /* MALI_TRUE if process is using a realtime scheduling policy */
+ mali_bool is_realtime;
+ /* The process priority in the range of OSK_PROCESS_PRIORITY_MIN
+ and OSK_PROCESS_PRIORITY_MAX. */
+ int priority;
+} osk_process_priority;
+
+/** @brief Check if the caller is using a realtime scheduling policy
+ *
+ * @return MALI_TRUE if process is running a realtime policy.
+ */
+OSK_STATIC_INLINE mali_bool osk_is_policy_realtime(void);
+
+/** @brief Retrieve the calling process priority and policy
+ *
+ * @param[out] prio structure to contain the process policy type
+ * and priority number
+ */
+OSK_STATIC_INLINE void osk_get_process_priority(osk_process_priority *prio);
+
+/** @} */ /* end group oskcredentials */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+/* pull in the arch header with the implementation */
+#include <osk/mali_osk_arch_credentials.h>
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _OSK_CREDENTIALS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _OSK_DEBUG_H_
+#define _OSK_DEBUG_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+#include <stdarg.h>
+#include <malisw/mali_malisw.h>
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @addtogroup oskdebug Debug
+ *
+ * OSK debug macros for asserts and debug messages. Mimics CDBG functionality.
+ *
+ * @{
+ */
+
+/**
+ * @brief OSK module IDs
+ */
+typedef enum
+{
+ OSK_UNKNOWN = 0, /**< @brief Unknown module */
+ OSK_OSK, /**< @brief ID of OSK module */
+ OSK_UKK, /**< @brief ID of UKK module */
+ OSK_BASE_MMU, /**< @brief ID of Base MMU */
+ OSK_BASE_JD, /**< @brief ID of Base Job Dispatch */
+ OSK_BASE_JM, /**< @brief ID of Base Job Manager */
+ OSK_BASE_CORE, /**< @brief ID of Base Core */
+ OSK_BASE_MEM, /**< @brief ID of Base Memory */
+ OSK_BASE_EVENT, /**< @brief ID of Base Event */
+ OSK_BASE_CTX, /**< @brief ID of Base Context */
+ OSK_BASE_PM, /**< @brief ID of Base Power Management */
+ OSK_UMP, /**< @brief ID of UMP module */
+ OSK_MODULES_ALL /**< @brief Select all the modules at once / Also gives the number of modules in the enum */
+} osk_module;
+
+/**
+ * Debug messages are sent to a particular channel (info, warn or error) or to all channels
+ */
+#define OSK_CHANNEL_INFO OSKP_CHANNEL_INFO /**< @brief No output*/
+#define OSK_CHANNEL_WARN OSKP_CHANNEL_WARN /**< @brief Standard output*/
+#define OSK_CHANNEL_ERROR OSKP_CHANNEL_ERROR /**< @brief Error output*/
+#define OSK_CHANNEL_RAW OSKP_CHANNEL_RAW /**< @brief Raw output*/
+#define OSK_CHANNEL_ALL OSKP_CHANNEL_ALL /**< @brief All the channels at the same time*/
+
+/** Function type that is called on an OSK_ASSERT() or OSK_ASSERT_MSG() */
+typedef void (osk_debug_assert_hook)( void * );
+
+typedef struct oskp_debug_assert_cb
+{
+ osk_debug_assert_hook *func;
+ void *param;
+} oskp_debug_assert_cb;
+
+/**
+ * @def OSK_DISABLE_ASSERTS
+ *
+ * @brief Indicates whether asserts are in use and evaluate their
+ * expressions. 0 indicates they are, any other value indicates that they are
+ * not.
+ */
+
+/**
+ * @def OSK_ASSERT_MSG(expr, ...)
+ * @brief Prints the given message if @a expr is false
+ *
+ * @note This macro does nothing if the flag @see OSK_DISABLE_ASSERTS is set to 1
+ *
+ * @param expr Boolean expression
+ * @param ... Message to display when @a expr is false, as a format string followed by format arguments.
+ * The format string and format arguments needs to be enclosed by parentheses.
+ * See oskp_validate_format_string for a list of supported format specifiers.
+ */
+#define OSK_ASSERT_MSG(expr, ...) OSKP_ASSERT_MSG(expr, __VA_ARGS__)
+
+/**
+ * @def OSK_ASSERT(expr)
+ * @brief Prints the expression @a expr if @a expr is false
+ *
+ * @note This macro does nothing if the flag @see OSK_DISABLE_ASSERTS is set to 1
+ *
+ * @param expr Boolean expression
+ */
+#define OSK_ASSERT(expr) OSKP_ASSERT(expr)
+
+/**
+ * @def OSK_INTERNAL_ASSERT(expr)
+ * @brief Asserts if @a expr is false.
+ * This assert function is for internal use of OSK functions which themselves are used to implement
+ * the OSK_ASSERT functionality. These functions should use OSK_INTERNAL_ASSERT which does not use
+ * any OSK functions to prevent ending up in a recursive loop.
+ *
+ * @note This macro does nothing if the flag @see OSK_DISABLE_ASSERTS is set to 1
+ *
+ * @param expr Boolean expression
+ */
+#define OSK_INTERNAL_ASSERT(expr) OSKP_INTERNAL_ASSERT(expr)
+
+/**
+ * @def OSK_DEBUG_CODE( X )
+ * @brief Executes the code inside the macro only in debug mode
+ *
+ * @param X Code to compile only in debug mode.
+ */
+#define OSK_DEBUG_CODE( X ) OSKP_DEBUG_CODE( X )
+
+/**
+ * @def OSK_PRINT(module, ...)
+ * @brief Prints given message
+ *
+ * Example:
+ * @code OSK_PRINT(OSK_BASE_MEM, " %d blocks could not be allocated", mem_allocated); @endcode will print:
+ * \n
+ * "10 blocks could not be allocated\n"
+ *
+ * @param module Name of the module which prints the message.
+ * @param ... Format string followed by a varying number of parameters
+ * See oskp_validate_format_string for a list of supported format specifiers.
+ */
+#define OSK_PRINT(module, ...) OSKP_PRINT_RAW(module, __VA_ARGS__)
+
+/**
+ * @def OSKP_PRINT_INFO(module, ...)
+ * @brief Prints "MALI<INFO,module_name>: " followed by the given message.
+ *
+ * Example:
+ * @code OSK_PRINT_INFO(OSK_BASE_MEM, " %d blocks could not be allocated", mem_allocated); @endcode will print:
+ * \n
+ * "MALI<INFO,BASE_MEM>: 10 blocks could not be allocated"\n
+ *
+ * @note Only gets compiled in for debug builds
+ *
+ * @param module Name of the module which prints the message.
+ * @param ... Format string followed by a varying number of parameters
+ * See oskp_validate_format_string for a list of supported format specifiers.
+ */
+#define OSK_PRINT_INFO(module, ...) OSKP_PRINT_INFO(module, __VA_ARGS__)
+
+/**
+ * @def OSK_PRINT_WARN(module, ...)
+ * @brief Prints "MALI<WARN,module_name>: " followed by the given message.
+ *
+ * Example:
+ * @code OSK_PRINT_WARN(OSK_BASE_MEM, " %d blocks could not be allocated", mem_allocated); @endcode will print:
+ * \n
+ * "MALI<WARN,BASE_MEM>: 10 blocks could not be allocated"\n
+ *
+ * @note Only gets compiled in for debug builds
+ *
+ * @param module Name of the module which prints the message.
+ * @param ... Format string followed by a varying number of parameters
+ * See oskp_validate_format_string for a list of supported format specifiers.
+ */
+#define OSK_PRINT_WARN(module, ...) OSKP_PRINT_WARN(module, __VA_ARGS__)
+
+/**
+ * @def OSK_PRINT_ERROR(module, ...)
+ * @brief Prints "MALI<ERROR,module_name>: " followed by the given message.
+ *
+ * Example:
+ * @code OSK_PRINT_ERROR(OSK_BASE_MEM, " %d blocks could not be allocated", mem_allocated); @endcode will print:
+ * \n
+ * "MALI<ERROR,BASE_MEM>: 10 blocks could not be allocated"\n
+ *
+ * @param module Name of the module which prints the message.
+ * @param ... Format string followed by a varying number of parameters
+ * See oskp_validate_format_string for a list of supported format specifiers.
+ */
+#define OSK_PRINT_ERROR(module, ...) OSKP_PRINT_ERROR(module, __VA_ARGS__)
+
+/**
+ * @def OSK_PRINT_ALLOW(module, channel)
+ * @brief Allow the given module to print on the given channel
+ * @note If @see OSK_USE_RUNTIME_CONFIG is disabled then this macro doesn't do anything
+ * @note Only gets compiled in for debug builds
+ * @param module is a @see osk_module
+ * @param channel is one of @see OSK_CHANNEL_INFO, @see OSK_CHANNEL_WARN, @see OSK_CHANNEL_ERROR,
+ * @see OSK_CHANNEL_ALL
+ * @return MALI_TRUE if the module is allowed to print on the channel.
+ */
+#define OSK_PRINT_ALLOW(module, channel) OSKP_PRINT_ALLOW(module, channel)
+
+/**
+ * @def OSK_PRINT_BLOCK(module, channel)
+ * @brief Prevent the given module from printing on the given channel
+ * @note If @see OSK_USE_RUNTIME_CONFIG is disabled then this macro doesn't do anything
+ * @note Only gets compiled in for debug builds
+ * @param module is a @see osk_module
+ * @param channel is one of @see OSK_CHANNEL_INFO, @see OSK_CHANNEL_WARN, @see OSK_CHANNEL_ERROR,
+ * @see OSK_CHANNEL_ALL
+ * @return MALI_TRUE if the module is allowed to print on the channel.
+ */
+#define OSK_PRINT_BLOCK(module, channel) OSKP_PRINT_BLOCK(module, channel)
+
+/**
+ * @brief Register a function to call on ASSERT
+ *
+ * Such functions will \b only be called during Debug mode, and for debugging
+ * features \b only. Do not rely on them to be called in general use.
+ *
+ * To disable the hook, supply NULL to \a func.
+ *
+ * @note This function is not thread-safe, and should only be used to
+ * register/deregister once in the module's lifetime.
+ *
+ * @param[in] func the function to call when an assert is triggered.
+ * @param[in] param the parameter to pass to \a func when calling it
+ */
+void osk_debug_assert_register_hook( osk_debug_assert_hook *func, void *param );
+
+/**
+ * @brief Call a debug assert hook previously registered with osk_debug_assert_register_hook()
+ *
+ * @note This function is not thread-safe with respect to multiple threads
+ * registering functions and parameters with
+ * osk_debug_assert_register_hook(). Otherwise, thread safety is the
+ * responsibility of the registered hook.
+ */
+void oskp_debug_assert_call_hook( void );
+
+/**
+ * @brief Convert a module id into a module name.
+ *
+ * @param module ID of the module to convert
+ * @note module names are stored in : @see oskp_str_modules.
+ * @return the name of the given module ID as a string of characters.
+ */
+const char* oskp_module_to_str(const osk_module module);
+
+/**
+ * @brief Validate the format string
+ *
+ * Validates the printf style format string against the formats
+ * that are supported by the OSK_PRINT macros. If an invalid
+ * format is used, a warning message is printed identifying
+ * the unsupported format specifier.
+ *
+ * Supported length and specifiers in the format string are:
+ *
+ * "d", "ld", "lld",
+ * "x", "lx", "llx",
+ * "X", "lX", "llX",
+ * "p",
+ * "c",
+ * "s"
+ *
+ * Notes:
+ * - in release builds this function does nothing.
+ * - this function takes a variable number of arguments to
+ * ease using it with variadic macros. Only the format
+ * argument is used though.
+ *
+ * @param format format string
+ *
+ */
+void oskp_validate_format_string(const char *format, ...);
+
+/**
+ * @brief printf-style string formatting.
+ *
+ * Refer to the cutils specification for restrictions on the format string.
+ *
+ * @param str output buffer
+ * @param size size of the output buffer in bytes (incl. eos)
+ * @param format the format string
+ * See oskp_validate_format_string for a list of supported
+ * format specifiers.
+ * @param [in] ... The variadic arguments
+ *
+ * @return The number of characters written on success, or a negative value
+ * on failure.
+ */
+s32 osk_snprintf(char *str, size_t size, const char *format, ...);
+
+/**
+ * @brief Get thread information for the current thread
+ *
+ * The information is for debug purposes only. For example, the current CPU for
+ * the thread could've changed by the time you access the returned information.
+ *
+ * On systems that support 64-bit thread IDs, the thread ID will be
+ * truncated. Therefore, this only gives an appoximate guide as to which thread
+ * is making the call.
+ *
+ * @param[out] thread_id first 32-bits of the current thread's ID
+ * @param[out] cpu_nr the CPU that the thread was probably executing on at the
+ * time of the call.
+ */
+OSK_STATIC_INLINE void osk_debug_get_thread_info( u32 *thread_id, u32 *cpu_nr );
+
+/* @} */ /* end group oskdebug */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+#include <osk/mali_osk_arch_debug.h>
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _OSK_DEBUG_H_ */
--- /dev/null
+/*
+ * * Copyright:
+ * ----------------------------------------------------------------------------
+ * This confidential and proprietary software may be used only as authorized
+ * by a licensing agreement from ARM Limited.
+ * (C) COPYRIGHT 2011 ARM Limited, ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorized copies and
+ * copies may only be made to the extent permitted by a licensing agreement
+ * from ARM Limited.
+ * ----------------------------------------------------------------------------
+ */
+
+#ifndef _OSK_FAILURE_H_
+#define _OSK_FAILURE_H_
+/** @file mali_osk_failure.h
+ *
+ * Base kernel side failure simulation mechanism interface.
+ *
+ * Provides a mechanism to simulate failure of
+ * functions which use the OSK_SIMULATE_FAILURE macro. This is intended to
+ * exercise error-handling paths during testing.
+ */
+#include <malisw/mali_malisw.h>
+#include "osk/include/mali_osk_debug.h"
+
+/**
+ * @addtogroup osk
+ * @{
+ */
+
+/**
+ * @addtogroup osk_failure Simulated failure
+ * @{
+ */
+
+/**
+ * @addtogroup osk_failure_public Public
+ * @{
+ */
+/**
+ * @brief Decide whether or not to simulate a failure in a given module
+ *
+ * Functions that can return a failure indication should use this macro to
+ * decide whether to do so in cases where no genuine failure occurred. This
+ * allows testing of error-handling paths in callers of those functions. A
+ * module ID must be specified to ensure that failures are only simulated in
+ * those modules for which they have been enabled.
+ *
+ * If it evaluates as MALI_TRUE, a message may be printed giving the location
+ * of the macro usage: the actual behavior is defined by @ref OSK_ON_FAIL.
+ *
+ * A break point set on the oskp_failure function will halt execution
+ * before this macro evaluates as MALI_TRUE.
+ *
+ * @param[in] module Numeric ID of the module using the macro
+ *
+ * @return MALI_FALSE if execution should continue as normal; otherwise
+ * a failure should be simulated by the code using this macro.
+ *
+ * @note Unless simulation of failures was enabled at compilation time, this
+ * macro always evaluates as MALI_FALSE.
+ */
+#if OSK_SIMULATE_FAILURES
+#define OSK_SIMULATE_FAILURE( module ) \
+ ( OSKP_SIMULATE_FAILURE_IS_ENABLED( (module), OSK_CHANNEL_INFO ) && \
+ oskp_is_failure_on() &&\
+ oskp_simulate_failure( module, OSKP_PRINT_TRACE, OSKP_PRINT_FUNCTION ) )
+#else
+#define OSK_SIMULATE_FAILURE( module ) \
+ ( CSTD_NOP( module ), MALI_FALSE )
+#endif
+
+
+/**
+ * @brief Get the number of potential failures
+ *
+ * This function can be used to find out the total number of potential
+ * failures during a test, before using @ref osk_set_failure_range to
+ * set the number of successes to allow. This allows testing of error-
+ * handling paths to be parallelized (in different processes) by sub-
+ * dividing the range of successes to allow before provoking a failure.
+ *
+ * @return The number of times the @ref OSK_SIMULATE_FAILURE macro has been
+ * evaluated since the counter was last reset.
+ */
+u64 osk_get_potential_failures( void );
+
+/**
+ * @brief Set the range of failures to simulate
+ *
+ * This function configures a range of potential failures to be tested by
+ * simulating actual failure. The @ref OSK_SIMULATE_FAILURE macro will
+ * evaluate as MALI_FALSE for the first @p start evaluations after the range
+ * is set; then as MALI_TRUE for the next @p end - @p start evaluations;
+ * finally, as MALI_FALSE after being evaluated @p end times (until the
+ * mechanism is reset). @p end must be greater than or equal to @p start.
+ *
+ * This function also resets the count of successes allowed so far.
+ *
+ * @param[in] start Number of potential failures to count before simulating
+ * the first failure, or U64_MAX to never fail.
+ * @param[in] end Number of potential failures to count before allowing
+ * resumption of success, or U64_MAX to fail all after
+ * @p first.
+ */
+void osk_set_failure_range( u64 start, u64 end );
+
+/**
+ * @brief Find out whether a failure was simulated
+ *
+ * This function can be used to find out whether an apparent failure was
+ * genuine or simulated by @ref OSK_SIMULATE_FAILURE macro.
+ *
+ * @return MALI_FALSE unless a failure was simulated since the last call to
+ * the @ref osk_set_failure_range function.
+ * @since 2.3
+ */
+mali_bool osk_failure_simulated( void );
+
+/** @} */
+/* end public*/
+
+/**
+ * @addtogroup osk_failure_private Private
+ * @{
+ */
+
+/**
+ * @brief Decide whether or not to simulate a failure
+ *
+ * @param[in] module Numeric ID of the module that can fail
+ * @param[in] trace Pointer to string giving the location in the source code
+ * @param[in] function Pointer to name of the calling function
+ *
+ * @return MALI_FALSE if execution should continue as normal; otherwise
+ * a failure should be simulated by the calling code.
+ */
+
+mali_bool oskp_simulate_failure( osk_module module,
+ const char *trace,
+ const char *function );
+mali_bool oskp_is_failure_on(void);
+void oskp_failure_init( void );
+void oskp_failure_term( void );
+/** @} */
+/* end osk_failure_private group*/
+
+/** @} */
+/* end osk_failure group*/
+
+/** @} */
+/* end osk group*/
+
+
+
+
+#endif /* _OSK_FAILURE_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_lists.h
+ * Implementation of the OS abstraction layer for the kernel device driver.
+ * Note that the OSK list implementation is copied from the CUTILS
+ * doubly linked list (DLIST) implementation.
+ */
+
+#ifndef _OSK_LISTS_H_
+#define _OSK_LISTS_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+#include <osk/mali_osk_common.h>
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @addtogroup osk_dlist Doubly-linked list
+ * @{
+ */
+/**
+ * @addtogroup osk_dlist_public Public
+ * @{
+ */
+/**
+ * @brief Item of a list
+ *
+ * @note Can be integrated inside a wider structure.
+ */
+typedef struct osk_dlist_item
+{
+ struct
+ {
+ struct osk_dlist_item *next; /**< @private */
+ struct osk_dlist_item *prev; /**< @private */
+ }oskp; /**< @private*/
+}osk_dlist_item;
+
+/**
+ * @brief Doubly-linked list
+ */
+typedef struct osk_dlist
+{
+ struct
+ {
+ struct osk_dlist_item *front; /**< @private */
+ struct osk_dlist_item *back; /**< @private */
+ }oskp; /**< @private*/
+}osk_dlist;
+
+/**
+ * @brief Test if @c container_ptr is the back of the list
+ *
+ * @param [in] container_ptr Pointer to the front of the container to test.
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @note An assert is triggered if @a container_ptr is NULL.
+ * @note If @c attribute is invalid then the behavior is undefined.
+ *
+ * @return Returns MALI_TRUE if @c container_ptr is the back of the list.
+ */
+#define OSK_DLIST_IS_BACK(container_ptr, attribute)\
+ (NULL == (OSK_CHECK_PTR(container_ptr))->attribute.oskp.next)
+
+/**
+ * @brief Test if @c container_ptr is the front of the list
+ *
+ * @param [in] container_ptr Pointer to the front of the container to test.
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @note An assert is triggered if @a container_ptr is NULL.
+ * @note If @c attribute is invalid then the behavior is undefined.
+ *
+ * @return Returns MALI_TRUE if @c container_ptr is the front of the list.
+ */
+#define OSK_DLIST_IS_FRONT(container_ptr, attribute)\
+ (NULL == (OSK_CHECK_PTR(container_ptr))->attribute.oskp.prev)
+
+/**
+ * @brief Test if @c container_ptr is valid
+ *
+ * @param [in] container_ptr Pointer to the front of the container to test.
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @note If @c attribute is invalid then the behavior is undefined.
+ *
+ * @return Returns MALI_TRUE if @c container_ptr is valid or MALI_FALSE otherwise.
+ */
+#define OSK_DLIST_IS_VALID(container_ptr, attribute)\
+ ( NULL != (container_ptr) )
+
+/**
+ * @brief Return the next item in the list
+ *
+ * @param [in] container_ptr Pointer to an item of type @c type
+ * @param [in] type Type of the container
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @return A pointer to the next container item, or @c NULL.
+
+ * @note If this macro evaluates as null then the back of the list has been reached.
+ * @note An assert is triggered if @a container_ptr is NULL.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+#define OSK_DLIST_NEXT(container_ptr, type, attribute)\
+ ( OSK_DLIST_IS_BACK( container_ptr, attribute ) ?\
+ NULL :CONTAINER_OF( (container_ptr)->attribute.oskp.next, type, attribute ) )
+
+/**
+ * @brief Return MALI_TRUE if the list is empty
+ *
+ * @param [in] osk_dlist_ptr Pointer to the @c osk_dlist to test.
+ *
+ * @note An assert is triggered if @a osk_dlist_ptr is NULL.
+ *
+ * @return Returns MALI_TRUE if @c osk_dlist_ptr is an empty list.
+ */
+#define OSK_DLIST_IS_EMPTY(osk_dlist_ptr)\
+ (NULL == OSK_CHECK_PTR(osk_dlist_ptr)->oskp.front)
+
+/**
+ * @brief Return the previous item in the list
+ *
+ * @param [in] container_ptr Pointer to an item of type @c type
+ * @param [in] type Type of the container
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @return A pointer to the previous container item, or @c NULL.
+
+ * @note If this macro evaluates as null then the front of the list has been reached.
+ * @note An assert is triggered if @a container_ptr is NULL.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+#define OSK_DLIST_PREV(container_ptr, type, attribute)\
+ ( OSK_DLIST_IS_FRONT( container_ptr, attribute ) ?\
+ NULL : CONTAINER_OF( (container_ptr)->attribute.oskp.prev, type, attribute) )
+
+/**
+ * @brief Return the front container of the list
+ *
+ * @param [in] osk_dlist_ptr Pointer to a list
+ * @param [in] type Type of the list container
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @return A pointer to the front container item, or @c NULL.
+
+ * @note If this macro evaluates as null then the list is empty.
+ * @note An assert is triggered if @a osk_dlist_ptr is NULL.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+#define OSK_DLIST_FRONT(osk_dlist_ptr, type, attribute)\
+ ( OSK_CHECK_PTR( osk_dlist_ptr )->oskp.front == NULL ?\
+ NULL : CONTAINER_OF( (osk_dlist_ptr)->oskp.front, type, attribute ) )
+
+/**
+ * @brief Check whether or not @c container_ptr is a member of @c osk_dlist_ptr.
+ *
+ * @param [in] osk_dlist_ptr Pointer to a list
+ * @param [in] container_ptr Pointer to the item to check.
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @return MALI_TRUE if @c container_ptr is a member of @c osk_dlist_ptr, MALI_FALSE if not.
+ *
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL.
+ * @note An assert is triggered if @c container_to_remove_ptr is NULL.
+ * @note If @c attribute is invalid then the behavior is undefined.
+ */
+#define OSK_DLIST_MEMBER_OF(osk_dlist_ptr, container_ptr, attribute)\
+ oskp_dlist_member_of(osk_dlist_ptr, &(OSK_CHECK_PTR(container_ptr))->attribute)
+
+/**
+ * @brief Return the back container of the list
+ *
+ * @param [in] osk_dlist_ptr Pointer to a list
+ * @param [in] type Type of the list container
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @return A pointer to the back container item, or @c NULL.
+ *
+ * @note If this macro evaluates as null then the list is empty.
+ * @note An assert is triggered if @a osk_dlist_ptr is NULL.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+#define OSK_DLIST_BACK(osk_dlist_ptr, type, attribute)\
+ ( OSK_CHECK_PTR( osk_dlist_ptr )->oskp.back == NULL ?\
+ NULL : CONTAINER_OF( (osk_dlist_ptr)->oskp.back, type, attribute) )
+
+/**
+ * @brief Initialize a list
+ *
+ * @param [out] osk_dlist_ptr Pointer to a osk_dlist
+ *
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL.
+ */
+#define OSK_DLIST_INIT(osk_dlist_ptr)\
+ do\
+ {\
+ OSK_CHECK_PTR(osk_dlist_ptr); \
+ (osk_dlist_ptr)->oskp.front = NULL; \
+ (osk_dlist_ptr)->oskp.back = NULL;\
+ }while(MALI_FALSE)
+
+/**
+ * @brief Append @c container_to_insert_ptr at the back of @c osk_dlist_ptr
+ *
+ * The front and the back of the list are automatically adjusted.
+ *
+ * @param [in, out] osk_dlist_ptr Pointer to a list
+ * @param [in, out] container_to_insert_ptr Pointer to an item to insert of type @c type.
+ * @param [in] type Type of the list container
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL.
+ * @note An assert is triggered if @c container_to_insert_ptr is NULL or if it already belongs to the list.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+#define OSK_DLIST_PUSH_BACK(osk_dlist_ptr, container_to_insert_ptr, type, attribute)\
+ OSK_DLIST_INSERT_BEFORE(osk_dlist_ptr, container_to_insert_ptr, NULL, type, attribute)
+
+/**
+ * @brief Insert @c container_to_insert_ptr at the front of @c osk_dlist_ptr
+ *
+ * The front and the back of the list are automatically adjusted.
+ *
+ * @param [in, out] osk_dlist_ptr Pointer to a list
+ * @param [in, out] container_to_insert_ptr Pointer to an item to insert of type @c type.
+ * @param [in] type Type of the list container
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL.
+ * @note An assert is triggered if @c container_to_insert_ptr is NULL or if it already belongs to the list.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+#define OSK_DLIST_PUSH_FRONT(osk_dlist_ptr, container_to_insert_ptr, type, attribute)\
+ OSK_DLIST_INSERT_AFTER(osk_dlist_ptr, container_to_insert_ptr, NULL, type, attribute)
+
+ /**
+ * @brief Remove the back of @c osk_dlist_ptr and return the element just removed
+ *
+ * The front and the back of the list are automatically adjusted.
+ *
+ * @param [in, out] osk_dlist_ptr Pointer to a list
+ * @param [in] type Type of the list container
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @return A pointer to a container item.
+ *
+ * @note If @c OSK_DLIST_IS_VALID returns MALI_FALSE when testing the returned pointer then the list is empty
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL or empty.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+#define OSK_DLIST_POP_BACK(osk_dlist_ptr, type, attribute)\
+ CONTAINER_OF(\
+ oskp_dlist_remove(\
+ osk_dlist_ptr, \
+ &OSK_CHECK_PTR( OSK_DLIST_BACK(osk_dlist_ptr, type, attribute) )->attribute), \
+ type, \
+ attribute)
+
+ /**
+ * @brief Remove the front of @c osk_dlist_ptr and return the element just removed
+ *
+ * The front and the back of the list are automatically adjusted.
+ *
+ * @note The list must contain at least one item.
+ *
+ * @param [in, out] osk_dlist_ptr Pointer to a list
+ * @param [in] type Type of the list container
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @return A pointer to a container item.
+ *
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL or empty.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+#define OSK_DLIST_POP_FRONT(osk_dlist_ptr, type, attribute)\
+ CONTAINER_OF(\
+ oskp_dlist_remove(\
+ osk_dlist_ptr, \
+ &OSK_CHECK_PTR( OSK_DLIST_FRONT(osk_dlist_ptr, type, attribute) )->attribute), \
+ type, \
+ attribute)
+
+/**
+ * @brief Append @c container_to_insert_ptr after @c container_pos_ptr in @c osk_dlist_ptr
+ *
+ * @note Insert the new element at the list front if @c container_pos_ptr is NULL.
+ *
+ * The front and the back of the list are automatically adjusted.
+ *
+ * @param [in, out] osk_dlist_ptr Pointer to a list
+ * @param [in, out] container_to_insert_ptr Pointer to an item to insert of type @c type.
+ * @param [in, out] container_pos_ptr Pointer to the item of type @c type after which inserting the new item.
+ * @param [in] type Type of the list container
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL.
+ * @note An assert is triggered if @c container_pos_ptr is not NULL and not a member of the list.
+ * @note An assert is triggered if @c container_to_insert_ptr is NULL or if it already belongs to the list.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+#define OSK_DLIST_INSERT_AFTER(osk_dlist_ptr, container_to_insert_ptr, container_pos_ptr, type, attribute)\
+ oskp_dlist_insert_after(\
+ osk_dlist_ptr, \
+ &(OSK_CHECK_PTR(container_to_insert_ptr))->attribute, \
+ &((type*)container_pos_ptr)->attribute, \
+ NULL == container_pos_ptr)
+/**
+ * @brief Append @c container_to_insert_ptr before @c container_pos_ptr in @c osk_dlist_ptr
+ *
+ * @note Insert the new element at the list back if @c container_pos_ptr is NULL.
+ *
+ * The front and the back of the list are automatically adjusted.
+ *
+ * @param [in, out] osk_dlist_ptr Pointer to a list
+ * @param [in, out] container_to_insert_ptr Pointer to an item to insert of type @c type.
+ * @param [in, out] container_pos_ptr Pointer to the item of type @c type before which inserting the new item.
+ * @param [in] type Type of the list container
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL.
+ * @note An assert is triggered if @c container_pos_ptr is not NULL and not a member of the list.
+ * @note An assert is triggered if @c container_to_insert_ptr is NULL or if it already belongs to the list.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+
+#define OSK_DLIST_INSERT_BEFORE(osk_dlist_ptr, container_to_insert_ptr, container_pos_ptr, type, attribute)\
+ oskp_dlist_insert_before(\
+ osk_dlist_ptr, \
+ &(OSK_CHECK_PTR(container_to_insert_ptr))->attribute, \
+ &((type*)container_pos_ptr)->attribute, \
+ NULL == container_pos_ptr)
+
+/**
+ * @brief Remove an item container from a doubly-linked list and return a pointer to the element
+ * which was next in the list.
+ *
+ * The front and the back of the list are automatically adjusted.
+ *
+ * @param [in, out] osk_dlist_ptr Pointer to a list
+ * @param [in, out] container_to_remove_ptr Pointer to an item to remove of type @c type.
+ * @param [in] type Type of the list container
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @return A pointer to the item container that was immediately after the one
+ * removed from the list, or @c NULL.
+ *
+ * @note If this macro evaluates as null then the back of the list has been reached.
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL.
+ * @note An assert is triggered if @c container_to_remove_ptr is NULL or if it doesn't belong to the list.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+
+ * @pre @p osk_dlist_ptr must have been initialized by @ref OSK_DLIST_INIT.
+ * @pre @p container_to_remove_ptr must be a member of list @p osk_dlist_ptr.
+ * @post @p container_to_remove_ptr is no longer a member of list @p osk_dlist_ptr.
+ *
+ */
+
+#define OSK_DLIST_REMOVE_AND_RETURN_NEXT(osk_dlist_ptr, container_to_remove_ptr, type, attribute)\
+ ( OSK_DLIST_IS_BACK( container_to_remove_ptr, attribute ) ?\
+ ( oskp_dlist_remove( osk_dlist_ptr, &( container_to_remove_ptr )->attribute ), NULL ) :\
+ CONTAINER_OF( oskp_dlist_remove_and_return_next( osk_dlist_ptr,\
+ &( container_to_remove_ptr )->attribute ),\
+ type,\
+ attribute ) )
+
+/**
+ * @brief Remove an item container from a doubly-linked list.
+ *
+ * The front and the back of the list are automatically adjusted.
+ *
+ * @param [in, out] osk_dlist_ptr Pointer to a list
+ * @param [in, out] container_to_remove_ptr Pointer to an item to remove of type @c type.
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @note An assert error is triggered if @c osk_dlist_ptr is NULL.
+ * @note An assert error is triggered if @c container_to_remove_ptr is NULL or if it doesn't belong to the list.
+ * @note If @c attribute is invalid then the behavior is undefined.
+ *
+ * @pre @p osk_dlist_ptr must have been initialized by @ref OSK_DLIST_INIT.
+ * @pre @p container_to_remove_ptr must be a member of list @p osk_dlist_ptr.
+ * @post @p container_to_remove_ptr is no longer a member of list @p osk_dlist_ptr.
+ */
+#define OSK_DLIST_REMOVE(osk_dlist_ptr, container_to_remove_ptr, attribute)\
+ oskp_dlist_remove_item(osk_dlist_ptr, &((OSK_CHECK_PTR(container_to_remove_ptr))->attribute) )
+
+/**
+ * @brief Remove an item container from a doubly-linked list and return a pointer to the element which was the
+ * previous one in the list.
+ *
+ * The front and the back of the list are automatically adjusted.
+ *
+ * @param [in, out] osk_dlist_ptr Pointer to a list
+ * @param [in, out] container_to_remove_ptr Pointer to an item to remove of type @c type.
+ * @param [in] type Type of the list container
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ *
+ * @return A pointer to the item container that was immediately before the one
+ * removed from the list, or @c NULL.
+ *
+ * @note If this macro evaluates as null then the front of the list has been reached.
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL.
+ * @note An assert is triggered if @c container_to_remove_ptr is NULL or if it doesn't belong to the list.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ *
+ * @pre @p osk_dlist_ptr must have been initialized by @ref OSK_DLIST_INIT.
+ * @pre @p container_to_remove_ptr must be a member of list @p osk_dlist_ptr.
+ * @post @p container_to_remove_ptr is no longer a member of list @p osk_dlist_ptr.
+ */
+
+#define OSK_DLIST_REMOVE_AND_RETURN_PREV(osk_dlist_ptr, container_to_remove_ptr, type, attribute)\
+ ( OSK_DLIST_IS_FRONT( container_to_remove_ptr, attribute ) ?\
+ ( oskp_dlist_remove( osk_dlist_ptr, &( container_to_remove_ptr )->attribute ), NULL ) :\
+ CONTAINER_OF( oskp_dlist_remove_and_return_prev( osk_dlist_ptr,\
+ &( container_to_remove_ptr )->attribute ),\
+ type,\
+ attribute ) )
+
+
+/**
+ * @brief Remove and call the destructor function for every item in the list, walking from start to end.
+ *
+ * @param [in, out] osk_dlist_ptr Pointer to the list to empty
+ * @param [in] type Type of the list container.
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ * @param [in] destructor_func Destructor function called for every item present in the list.
+ *
+ * This function has to be of the form void func(type* item);
+ *
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL.
+ * @note An assert is triggered if @c destructor_func is NULL.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+#define OSK_DLIST_EMPTY_LIST(osk_dlist_ptr, type, attribute, destructor_func)\
+ do\
+ {\
+ type* oskp_it;\
+ OSK_ASSERT(NULL != osk_dlist_ptr); \
+ OSK_ASSERT(NULL != destructor_func); \
+ oskp_it = OSK_DLIST_FRONT(osk_dlist_ptr, type, attribute);\
+ while ( oskp_it != NULL )\
+ {\
+ type* to_delete = oskp_it;\
+ oskp_it = OSK_DLIST_REMOVE_AND_RETURN_NEXT(osk_dlist_ptr, oskp_it, type, attribute);\
+ destructor_func(to_delete);\
+ }\
+ }while(MALI_FALSE)
+
+/**
+ * @brief Remove and call the destructor function for every item in the list, walking from the end and to the front.
+ *
+ * @param [in, out] osk_dlist_ptr Pointer to the list to empty
+ * @param [in] type Type of the list container.
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ * @param [in] destructor_func Destructor function called for every item present in the list.
+ *
+ * This function has to be of the form void func(type* item);
+ *
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL.
+ * @note An assert is triggered if @c destructor_func is NULL.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+
+#define OSK_DLIST_EMPTY_LIST_REVERSE(osk_dlist_ptr, type, attribute, destructor_func)\
+ do\
+ {\
+ type* oskp_it;\
+ OSK_ASSERT(NULL != osk_dlist_ptr); \
+ OSK_ASSERT(NULL != destructor_func); \
+ oskp_it = OSK_DLIST_BACK(osk_dlist_ptr, type, attribute);\
+ while ( oskp_it != NULL )\
+ {\
+ type* to_delete = oskp_it;\
+ oskp_it = OSK_DLIST_REMOVE_AND_RETURN_PREV(osk_dlist_ptr, oskp_it, type, attribute);\
+ destructor_func(to_delete);\
+ }\
+ }while(MALI_FALSE)
+
+
+
+/**
+ * @brief Iterate forward through each container item of the given list
+ *
+ * @param [in, out] osk_dlist_ptr Pointer to a list
+ * @param [in] type Container type of the list
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ * @param [out] container_iterator Iterator variable of type "type*" to use to iterate through the list.
+ *
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+#define OSK_DLIST_FOREACH(osk_dlist_ptr, type, attribute, container_iterator)\
+ OSK_ASSERT(NULL != osk_dlist_ptr); \
+ for(\
+ container_iterator = OSK_DLIST_FRONT(osk_dlist_ptr, type, attribute);\
+ NULL != container_iterator; \
+ container_iterator = OSK_DLIST_NEXT(container_iterator, type, attribute))
+
+/**
+ * @brief Reverse iterate through each container item of the given list
+ *
+ * @param [in, out] osk_dlist_ptr Pointer to a list
+ * @param [in] type Container type of the list
+ * @param [in] attribute Attribute of the container of type @c osk_dlist_item
+ * @param [out] container_iterator Iterator variable of type "type*" to use to iterate through the list.
+ *
+ * @note An assert is triggered if @c osk_dlist_ptr is NULL.
+ * @note If @c type or @c attribute is invalid then the behavior is undefined.
+ */
+#define OSK_DLIST_FOREACH_REVERSE(osk_dlist_ptr, type, attribute, container_iterator)\
+ OSK_ASSERT(NULL != osk_dlist_ptr); \
+ for(\
+ container_iterator = OSK_DLIST_BACK(osk_dlist_ptr, type, attribute);\
+ NULL != container_iterator; \
+ container_iterator = OSK_DLIST_PREV(container_iterator, type, attribute))
+
+/**
+ * @}
+ */
+/* End osk_dlist_public */
+/**
+ * @addtogroup osk_dlist_private Private
+ * @{
+ */
+
+/**
+ * @brief Insert a new item after an existing one.
+ *
+ * @param [in, out] list_ptr Pointer to the list the new item is going to be added to.
+ * @param [in, out] item_to_insert New item to insert in the list.
+ * @param [in, out] position Position after which to add the new item.
+ * @param [in] insert_at_front If this argument is equal to MALI_TRUE then @c position is ignored and the
+ * new item is added to the front.
+ *
+ * @note An assert is triggered if @c list_ptr is NULL.
+ * @note An assert is triggered if @c insert_at_front is MALI_FALSE and @c position is NULL.
+ */
+OSK_STATIC_INLINE void oskp_dlist_insert_after(osk_dlist * const list_ptr, osk_dlist_item * const item_to_insert,
+ osk_dlist_item * const position, const mali_bool insert_at_front);
+
+/**
+ * @brief Insert a new item before an existing one.
+ *
+ * @param [in, out] list_ptr Pointer to the list the new item is going to be added to.
+ * @param [in, out] item_to_insert New item to insert in the list.
+ * @param [in, out] position Position before which to add the new item.
+ * @param [in] insert_at_back If this argument is equal to MALI_TRUE then @c position is ignored and the new
+ * item is added to the back
+ *
+ * @note An assert is triggered if @c list_ptr is NULL.
+ * @note An assert is triggered if @c insert_at_back is MALI_FALSE and @c position is NULL.
+ */
+
+OSK_STATIC_INLINE void oskp_dlist_insert_before(osk_dlist * const list_ptr, osk_dlist_item* const item_to_insert,
+ osk_dlist_item * const position, const mali_bool insert_at_back);
+
+/**
+ * @brief Remove a given item from the list and return the item which was next in the list
+ *
+ * @param [in, out] list_ptr List from which the item needs to be removed
+ * @param [in, out] item_to_remove Item to remove from the list
+ *
+ * @return A pointer to the item which was next in the list. Return NULL if the back has just been removed.
+ *
+ * @note An assert is triggered if @c list_ptr is NULL.
+ * @note An assert is triggered if @c item_to_remove is not a member of @c list_ptr
+
+ */
+OSK_STATIC_INLINE osk_dlist_item* oskp_dlist_remove_and_return_next(osk_dlist * const list_ptr,
+ osk_dlist_item * const item_to_remove) CHECK_RESULT;
+
+/**
+ * @brief Remove a given item from the list and return the item which was before in the list
+ *
+ * @param [in, out] list_ptr List from which the item needs to be removed
+ * @param [in, out] item_to_remove Item to remove from the list
+ *
+ * @return A pointer to the item which was before in the list. Return NULL if the front has just been removed.
+ *
+ * @note An assert is triggered if @c list_ptr is NULL.
+ * @note An assert is triggered if @c item_to_remove is not a member of @c list_ptr
+ */
+OSK_STATIC_INLINE osk_dlist_item* oskp_dlist_remove_and_return_prev(osk_dlist * const list_ptr,
+ osk_dlist_item * const item_to_remove) CHECK_RESULT;
+
+/**
+ * @brief Remove a given item from the list and return it.
+ *
+ * @param [in, out] list_ptr List from which the item needs to be removed
+ * @param [in, out] item_to_remove Item to remove from the list
+ *
+ * @return A pointer to the item which has been removed from the list.
+ *
+ * @note An assert is triggered if @c list_ptr is NULL.
+ * @note An assert is triggered if @c item_to_remove is not a member of @c list_ptr
+ */
+
+OSK_STATIC_INLINE osk_dlist_item* oskp_dlist_remove(osk_dlist * const list_ptr,
+ osk_dlist_item * const item_to_remove);
+
+/**
+ * @brief Check that @c item is a member of the @c list
+ *
+ * @param [in] list Metadata of the list
+ * @param [in] item Item to check
+ *
+ * @note An assert error is triggered if @c list is NULL.
+ *
+ * @return MALI_TRUE if @c item is a member of @c list or MALI_FALSE otherwise.
+ */
+OSK_STATIC_INLINE mali_bool oskp_dlist_member_of(const osk_dlist* const list, const osk_dlist_item* const item) CHECK_RESULT;
+
+/**
+ * @brief remove @c item_to_remove from @c front
+ *
+ * @param [in, out] front List from which the item needs to be removed
+ * @param [in, out] item_to_remove Item to remove from the list.
+ *
+ * @note An assert is triggered if @c list_ptr is NULL.
+ * @note An assert is triggered if @c item_to_remove is not a member of @c list_ptr
+ */
+OSK_STATIC_INLINE void oskp_dlist_remove_item(osk_dlist* const front, osk_dlist_item* const item_to_remove);
+
+/**
+ * @}
+ */
+/* end osk_dlist_private */
+/**
+ * @}
+ */
+/* end osk_dlist group */
+
+/**
+ * @addtogroup osk_dlist Doubly-linked list
+ * @{
+ */
+/**
+ * @addtogroup osk_dlist_private Private
+ * @{
+ */
+
+CHECK_RESULT OSK_STATIC_INLINE mali_bool oskp_dlist_member_of(const osk_dlist* const list, const osk_dlist_item* const item)
+{
+ mali_bool return_value = MALI_FALSE;
+ const osk_dlist_item* it;
+
+ OSK_ASSERT(NULL != list);
+
+ it = list->oskp.front;
+ while(NULL != it)
+ {
+ if(item == it)
+ {
+ return_value = MALI_TRUE;
+ break;
+ }
+
+ it = it->oskp.next;
+ }
+ return return_value;
+}
+
+OSK_STATIC_INLINE void oskp_dlist_insert_before(osk_dlist * const front, osk_dlist_item * const item_to_insert,
+ osk_dlist_item * const position, const mali_bool insert_at_back)
+{
+ OSK_ASSERT(NULL != front);
+ OSK_ASSERT(NULL != item_to_insert);
+ OSK_ASSERT((insert_at_back == MALI_TRUE) || (NULL != position));
+ OSK_ASSERT(MALI_FALSE == oskp_dlist_member_of(front, item_to_insert));
+
+ if(insert_at_back)
+ {
+ item_to_insert->oskp.prev = front->oskp.back;
+
+ /*if there are some other items in the list, update their links.*/
+ if(NULL != front->oskp.back)
+ {
+ front->oskp.back->oskp.next = item_to_insert;
+ }
+ item_to_insert->oskp.next = NULL;
+ front->oskp.back = item_to_insert;
+ }
+ else
+ {
+ /* insertion at a position which is not the back*/
+ OSK_ASSERT(MALI_FALSE != oskp_dlist_member_of(front, position));
+
+ item_to_insert->oskp.prev = position->oskp.prev;
+ item_to_insert->oskp.next = position;
+ position->oskp.prev = item_to_insert;
+
+ /*if there are some other items in the list, update their links.*/
+ if(NULL != item_to_insert->oskp.prev)
+ {
+ item_to_insert->oskp.prev->oskp.next = item_to_insert;
+ }
+
+ }
+
+ /* Did the element inserted became the new front */
+ if(front->oskp.front == item_to_insert->oskp.next)
+ {
+ front->oskp.front = item_to_insert;
+ }
+}
+
+OSK_STATIC_INLINE
+void oskp_dlist_insert_after(osk_dlist * const front, osk_dlist_item * const item_to_insert,
+ osk_dlist_item * const position, mali_bool insert_at_front)
+{
+ OSK_ASSERT(NULL != front);
+ OSK_ASSERT(NULL != item_to_insert);
+ OSK_ASSERT((insert_at_front == MALI_TRUE) || (NULL != position));
+ OSK_ASSERT(MALI_FALSE == oskp_dlist_member_of(front, item_to_insert));
+
+ if(insert_at_front)
+ {
+ item_to_insert->oskp.next = front->oskp.front;
+
+ /*if there are some other items in the list, update their links.*/
+ if(NULL != front->oskp.front)
+ {
+ front->oskp.front->oskp.prev = item_to_insert;
+ }
+ item_to_insert->oskp.prev = NULL;
+ front->oskp.front = item_to_insert;
+ }
+ else
+ {
+ /* insertion at a position which is not the front */
+ OSK_ASSERT(MALI_FALSE != oskp_dlist_member_of(front, position));
+
+ item_to_insert->oskp.next = position->oskp.next;
+ item_to_insert->oskp.prev = position;
+ position->oskp.next = item_to_insert;
+
+ /*if the item has not been inserted at the back, then update the links of the next item*/
+ if(NULL != item_to_insert->oskp.next)
+ {
+ item_to_insert->oskp.next->oskp.prev = item_to_insert;
+ }
+ }
+
+ /* Is the item inserted the new back ?*/
+ if(front->oskp.back == item_to_insert->oskp.prev)
+ {
+ front->oskp.back = item_to_insert;
+ }
+}
+
+OSK_STATIC_INLINE
+void oskp_dlist_remove_item(osk_dlist* const front, osk_dlist_item* const item_to_remove)
+{
+ OSK_ASSERT(NULL != front);
+ OSK_ASSERT(NULL != item_to_remove);
+ OSK_ASSERT(MALI_TRUE == oskp_dlist_member_of(front, item_to_remove));
+
+ /* if the item to remove is the current front*/
+ if( front->oskp.front == item_to_remove )
+ {
+ /* then make the front point to the next item*/
+ front->oskp.front = item_to_remove->oskp.next;
+ }
+ else
+ {
+ /* else just the previous item point to the next one*/
+ item_to_remove->oskp.prev->oskp.next = item_to_remove->oskp.next;
+ }
+
+ /* if the item to remove is the current back*/
+ if(front->oskp.back == item_to_remove)
+ {
+ /* then make the back point to the previous item*/
+ front->oskp.back = item_to_remove->oskp.prev;
+ }
+ else
+ {
+ /* else just the next item point to the previous one*/
+ item_to_remove->oskp.next->oskp.prev = item_to_remove->oskp.prev;
+ }
+
+ item_to_remove->oskp.next = NULL;
+ item_to_remove->oskp.prev = NULL;
+}
+
+OSK_STATIC_INLINE
+osk_dlist_item* oskp_dlist_remove(osk_dlist * const front, osk_dlist_item * const item_to_remove)
+{
+ oskp_dlist_remove_item(front, item_to_remove);
+
+ item_to_remove->oskp.next = NULL;
+ item_to_remove->oskp.prev = NULL;
+
+ return item_to_remove;
+}
+
+
+CHECK_RESULT OSK_STATIC_INLINE
+osk_dlist_item* oskp_dlist_remove_and_return_next(osk_dlist * const front,
+ osk_dlist_item * const item_to_remove)
+{
+ osk_dlist_item *next;
+
+ OSK_ASSERT(NULL != front);
+ OSK_ASSERT(NULL != item_to_remove);
+
+ next = item_to_remove->oskp.next;
+ oskp_dlist_remove_item(front, item_to_remove);
+ return next;
+}
+
+CHECK_RESULT OSK_STATIC_INLINE
+osk_dlist_item* oskp_dlist_remove_and_return_prev(osk_dlist * const front,
+ osk_dlist_item * const item_to_remove)
+{
+ osk_dlist_item *prev;
+
+ OSK_ASSERT(NULL != front);
+ OSK_ASSERT(NULL != item_to_remove);
+
+ prev = item_to_remove->oskp.prev;
+ oskp_dlist_remove_item(front, item_to_remove);
+ return prev;
+}
+
+/**
+ * @}
+ */
+/* end osk_dlist_private */
+
+/**
+ * @}
+ */
+/* end osk_dlist group */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _OSK_LISTS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _OSK_LOCK_ORDER_H_
+#define _OSK_LOCK_ORDER_H_
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @addtogroup oskmutex_lockorder
+ * @{
+ */
+
+/**
+ * @anchor oskmutex_lockorder
+ * @par Lock ordering for Mutexes and Spinlocks
+ *
+ * When an OSK Rwlock, Mutex or Spinlock is initialized, it is given a locking order.
+ * This is a number that is checked in QA builds to detect possible deadlock
+ * conditions. The order is checked when a thread calls
+ * osk_rwlock_read_lock() / osk_rwlock_write_lock() / osk_mutex_lock() /
+ * osk_spinlock_lock() / osk_spinlock_irq_lock(). If the calling
+ * thread already holds a lock with an order less than that of the object being
+ * locked, an assertion failure will occur.
+ *
+ * Lock ordering must be respected between OSK Rwlocks, Mutexes, and Spinlocks.
+ * That is, when obtaining an OSK Rwlock, Mutex or Spinlock, its lock order
+ * must be lower than any other OSK Rwlock, Mutex or Spinlock held by the current thread.
+ *
+ */
+/** @{ */
+
+typedef enum
+{
+ /**
+ * Reserved mutex order, indicating that the mutex will be the last to be
+ * locked, and all other OSK mutexes are obtained before this one.
+ *
+ * All other lock orders must be after this one, because we use this to
+ * ASSERT that lock orders are >= OSK_LOCK_ORDER_LAST
+ */
+ OSK_LOCK_ORDER_LAST = 0,
+
+ /**
+ * Lock order for umpp_descriptor_mapping.
+ *
+ * This lock is always obtained last: no other locks are obtained whilst
+ * operating on a descriptor mapping, and so this should be as high as
+ * possible in this enum (lower in number) than any other lock held by UMP.
+ *
+ * It can have the same order as any other lock in UMP that is always
+ * obtained last.
+ */
+ OSK_LOCK_ORDER_UMP_DESCRIPTOR_MAPPING,
+
+ /**
+ * Lock order for mutex protecting umpp_device::secure_id_map (this is in
+ * the 'single global UMP device').
+ *
+ * This must be obtained after (lower in number than) the
+ * OSK_LOCK_ORDER_UMP_SESSION_LOCK, since the allocation is often looked up
+ * in secure_id_map while manipulating the umpp_session::memory_usage list.
+ */
+ OSK_LOCK_ORDER_UMP_IDMAP_LOCK,
+
+ /**
+ * Lock order for mutex protecting the umpp_session::memory_usage list
+ */
+ OSK_LOCK_ORDER_UMP_SESSION_LOCK,
+
+
+ /**
+ *
+ */
+ OSK_LOCK_ORDER_OSK_FAILURE,
+
+ /**
+ * For the power management metrics system
+ */
+ OSK_LOCK_ORDER_PM_METRICS,
+
+ /**
+ * For fast queue management, with very little processing and
+ * no other lock held within the critical section.
+ */
+ OSK_LOCK_ORDER_QUEUE = OSK_LOCK_ORDER_PM_METRICS,
+
+ /**
+ * For register trace buffer access in kernel space
+ */
+
+ OSK_LOCK_ORDER_TB,
+
+ /**
+ * For KBASE_TRACE_ADD<...> macros
+ */
+ OSK_LOCK_ORDER_TRACE,
+
+ /**
+ * For modification of the MMU mask register, which is done as a read-modify-write
+ */
+ OSK_LOCK_ORDER_MMU_MASK,
+ /**
+ * For access and modification to the power state of a device
+ */
+ OSK_LOCK_ORDER_POWER_MGMT = OSK_LOCK_ORDER_MMU_MASK,
+
+ /**
+ * For access to active_count in kbase_pm_device_data
+ */
+ OSK_LOCK_ORDER_POWER_MGMT_ACTIVE = OSK_LOCK_ORDER_POWER_MGMT,
+
+ /**
+ * For access to gpu_cycle_counter_requests in kbase_pm_device_data
+ */
+ OSK_LOCK_ORDER_POWER_MGMT_GPU_CYCLE_COUNTER,
+ /**
+ * For the resources used during MMU pf or low-level job handling
+ */
+ OSK_LOCK_ORDER_JS_RUNPOOL_IRQ,
+
+ /**
+ * For job slot management
+ *
+ * This is an IRQ lock, and so must be held after all sleeping locks
+ */
+ OSK_LOCK_ORDER_JSLOT,
+
+ /**
+ * For hardware counters collection setup
+ */
+ OSK_LOCK_ORDER_HWCNT,
+
+ /**
+ * For use when zapping a context (see kbase_jd_zap_context)
+ */
+ OSK_LOCK_ORDER_JD_ZAP_CONTEXT,
+
+ /**
+ * AS lock, used to access kbase_as structure.
+ *
+ * This must be held after:
+ * - Job Scheduler Run Pool lock (OSK_LOCK_ORDER_RUNPOOL)
+ *
+ * This is an IRQ lock, and so must be held after all sleeping locks
+ *
+ * @since OSU 1.9
+ */
+ OSK_LOCK_ORDER_AS,
+
+ /**
+ * Job Scheduling Run Pool lock
+ *
+ * This must be held after:
+ * - Job Scheduling Context Lock (OSK_LOCK_ORDER_JS_CTX)
+ * - Job Slot management lock (OSK_LOCK_ORDER_JSLOT)
+ *
+ * This is an IRQ lock, and so must be held after all sleeping locks
+ *
+ */
+ OSK_LOCK_ORDER_JS_RUNPOOL,
+
+
+ /**
+ * Job Scheduling Policy Queue lock
+ *
+ * This must be held after Job Scheduling Context Lock (OSK_LOCK_ORDER_JS_CTX).
+ *
+ * Currently, there's no restriction on holding this at the same time as the JSLOT/JS_RUNPOOL locks - but, this doesn't happen anyway.
+ *
+ */
+ OSK_LOCK_ORDER_JS_QUEUE,
+
+ /**
+ * Job Scheduling Context Lock
+ *
+ * This must be held after Job Dispatch lock (OSK_LOCK_ORDER_JCTX), but before:
+ * - The Job Slot lock (OSK_LOCK_ORDER_JSLOT)
+ * - The Run Pool lock (OSK_LOCK_ORDER_JS_RUNPOOL)
+ * - The Policy Queue lock (OSK_LOCK_ORDER_JS_QUEUE)
+ *
+ * In addition, it must be held before the VM Region Lock (OSK_LOCK_ORDER_MEM_REG),
+ * because at some point need to modify the MMU registers to update the address
+ * space on scheduling in the context.
+ *
+ */
+ OSK_LOCK_ORDER_JS_CTX,
+
+ /**
+ * For memory mapping management
+ */
+ OSK_LOCK_ORDER_MEM_REG,
+
+ /**
+ * For job dispatch management
+ */
+ OSK_LOCK_ORDER_JCTX,
+
+ /**
+ * Register queue lock for model
+ */
+ OSK_LOCK_ORDER_BASE_REG_QUEUE,
+
+ /**
+ * Reserved mutex order, indicating that the mutex will be the first to be
+ * locked, and all other OSK mutexes are obtained after this one.
+ *
+ * All other lock orders must be before this one, because we use this to
+ * ASSERT that lock orders are <= OSK_LOCK_ORDER_FIRST
+ */
+ OSK_LOCK_ORDER_FIRST
+} osk_lock_order;
+
+/** @} */
+
+/** @} */ /* end group oskmutex_lockorder */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _OSK_LOCK_ORDER_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _OSK_LOCKS_H_
+#define _OSK_LOCKS_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @defgroup osklocks Mutual Exclusion
+ *
+ * A read/write lock (rwlock) is used to control access to a shared resource,
+ * where multiple threads are allowed to read from the shared resource, but
+ * only one thread is allowed to write to the shared resources at any one time.
+ * A thread must specify the type of access (read/write) when locking the
+ * rwlock. If a rwlock is locked for write access, other threads that attempt
+ * to lock the same rwlock will block. If a rwlock is locked for read access,
+ * threads that attempts to lock the rwlock for write access, will block until
+ * until all threads with read access have unlocked the rwlock.
+
+ * @note If an OS does not provide a synchronisation object to implement a
+ * rwlock, a OSK mutex can be used instead for its implementation. This would
+ * only allow one reader or writer to access the shared resources at any one
+ * time.
+ *
+ * A mutex is used to control access to a shared resource, where only one
+ * thread is allowed access at any one time. A thread must lock the mutex
+ * to gain access; other threads that attempt to lock the same mutex will
+ * block. Mutexes can only be unlocked by the thread that holds the lock.
+ *
+ * @note OSK mutexes are intended for use in a situation where access to the
+ * shared resource is likely to be contended. OSK mutexes make use of the
+ * mutual exclusion primitives provided by the target OS, which often
+ * are considered "heavyweight".
+ *
+ * Spinlocks are also used to control access to a shared resource and
+ * enforce that only one thread has access at any one time. They differ from
+ * OSK mutexes in that they poll the mutex to obtain the lock. This makes a
+ * spinlock especially suited for contexts where you are not allowed to block
+ * while waiting for access to the shared resource. A OSK mutex could not be
+ * used in such a context as it can block while trying to obtain the mutex.
+ *
+ * A spinlock should be held for the minimum time possible, as in the contended
+ * case threads will not sleep but poll and therefore use CPU-cycles.
+ *
+ * While holding a spinlock, you must not sleep. You must not obtain a rwlock,
+ * mutex or do anything else that might block your thread. This is to prevent another
+ * thread trying to lock the same spinlock while your thread holds the spinlock,
+ * which could take a very long time (as it requires your thread to get scheduled
+ * in again and unlock the spinlock) or could even deadlock your system.
+ *
+ * Spinlocks are considered 'lightweight': for the uncontended cases, the mutex
+ * can be obtained quickly. For the lightly-contended cases on Multiprocessor
+ * systems, the mutex can be obtained quickly without resorting to
+ * "heavyweight" OS primitives.
+ *
+ * Two types of spinlocks are provided. A type that is safe to use when sharing
+ * a resource with an interrupt service routine, and one that should only be
+ * used to share the resource between threads. The former should be used to
+ * prevent deadlock between a thread that holds a spinlock while an
+ * interrupt occurs and the interrupt service routine trying to obtain the same
+ * spinlock too.
+ *
+ * @anchor oskmutex_spinlockdetails
+ * @par Important details of OSK Spinlocks.
+ *
+ * OSK spinlocks are not intended for high-contention cases. If high-contention
+ * usecases occurs frequently for a particular spinlock, then it is wise to
+ * consider using an OSK Mutex instead.
+ *
+ * @note An especially important reason for not using OSK Spinlocks in highly
+ * contended cases is that they defeat the OS's Priority Inheritance mechanisms
+ * that would normally alleviate Priority Inversion problems. This is because
+ * once the spinlock is obtained, the OS usually does not know which thread has
+ * obtained the lock, and so cannot know which thread must have its priority
+ * boosted to alleviate the Priority Inversion.
+ *
+ * As a guide, use a spinlock when CPU-bound for a short period of time
+ * (thousands of cycles). CPU-bound operations include reading/writing of
+ * memory or registers. Do not use a spinlock when IO bound (e.g. user input,
+ * buffered IO reads/writes, calls involving significant device driver IO
+ * calls).
+ */
+/** @{ */
+
+/**
+ * @brief Initialize a mutex
+ *
+ * Initialize a mutex structure. If the function returns successfully, the
+ * mutex is in the unlocked state.
+ *
+ * The caller must allocate the memory for the @see osk_mutex
+ * structure, which is then populated within this function. If the OS-specific
+ * mutex referenced from the structure cannot be initialized, an error is
+ * returned.
+ *
+ * The mutex must be terminated when no longer required, by using
+ * osk_mutex_term(). Otherwise, a resource leak may result in the OS.
+ *
+ * The mutex is initialized with a lock order parameter, \a order. Refer to
+ * @see oskmutex_lockorder for more information on Rwlock/Mutex/Spinlock lock
+ * ordering.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * It is a programming error to attempt to initialize a mutex that is
+ * currently initialized.
+ *
+ * @param[out] lock pointer to an uninitialized mutex structure
+ * @param[in] order the locking order of the mutex
+ * @return OSK_ERR_NONE on success, any other value indicates a failure.
+ */
+OSK_STATIC_INLINE osk_error osk_mutex_init(osk_mutex * const lock, osk_lock_order order) CHECK_RESULT;
+
+/**
+ * @brief Terminate a mutex
+ *
+ * Terminate the mutex pointed to by \a lock, which must be
+ * a pointer to a valid unlocked mutex. When the mutex is terminated, the
+ * OS-specific mutex is freed.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * It is a programming error to attempt to terminate a mutex that is currently
+ * terminated.
+ *
+ * @illegal It is illegal to call osk_mutex_term() on a locked mutex.
+ *
+ * @param[in] lock pointer to a valid mutex structure
+ */
+OSK_STATIC_INLINE void osk_mutex_term(osk_mutex * lock);
+
+/**
+ * @brief Lock a mutex
+ *
+ * Lock the mutex pointed to by \a lock. If the mutex is currently unlocked,
+ * the calling thread returns with the mutex locked. If a second thread
+ * attempts to lock the same mutex, it blocks until the first thread
+ * unlocks the mutex. If two or more threads are blocked waiting on the first
+ * thread to unlock the mutex, it is undefined as to which thread is unblocked
+ * when the first thread unlocks the mutex.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * It is a programming error to lock a mutex or spinlock with an order that is
+ * higher than any mutex or spinlock held by the current thread. Mutexes and
+ * spinlocks must be locked in the order of highest to lowest, to prevent
+ * deadlocks. Refer to @see oskmutex_lockorder for more information.
+ *
+ * It is a programming error to exit a thread while it has a locked mutex.
+ *
+ * It is a programming error to lock a mutex from an ISR context. In an ISR
+ * context you are not allowed to block what osk_mutex_lock() potentially does.
+ *
+ * @illegal It is illegal to call osk_mutex_lock() on a mutex that is currently
+ * locked by the caller thread. That is, it is illegal for the same thread to
+ * lock a mutex twice, without unlocking it in between.
+ *
+ * @param[in] lock pointer to a valid mutex structure
+ */
+OSK_STATIC_INLINE void osk_mutex_lock(osk_mutex * lock);
+
+/**
+ * @brief Unlock a mutex
+ *
+ * Unlock the mutex pointed to by \a lock. The calling thread must be the
+ * same thread that locked the mutex. If no other threads are waiting on the
+ * mutex to be unlocked, the function returns immediately, with the mutex
+ * unlocked. If one or more threads are waiting on the mutex to be unlocked,
+ * then this function returns, and a thread waiting on the mutex can be
+ * unblocked. It is undefined as to which thread is unblocked.
+ *
+ * @note It is not defined \em when a waiting thread is unblocked. For example,
+ * a thread calling osk_mutex_unlock() followed by osk_mutex_lock() may (or may
+ * not) obtain the lock again, preventing other threads from being
+ * released. Neither the 'immediately releasing', nor the 'delayed releasing'
+ * behavior of osk_mutex_unlock() can be relied upon. If such behavior is
+ * required, then you must implement it yourself, such as by using a second
+ * synchronization primitive.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * @illegal It is illegal for a thread to call osk_mutex_unlock() on a mutex
+ * that it has not locked, even if that mutex is currently locked by another
+ * thread. That is, it is illegal for any thread other than the 'owner' of the
+ * mutex to unlock it. And, you must not unlock an already unlocked mutex.
+ *
+ * @param[in] lock pointer to a valid mutex structure
+ */
+OSK_STATIC_INLINE void osk_mutex_unlock(osk_mutex * lock);
+
+/**
+ * @brief Initialize a spinlock
+ *
+ * Initialize a spinlock. If the function returns successfully, the
+ * spinlock is in the unlocked state.
+ *
+ * @note If the spinlock is used for sharing a resource with an interrupt service
+ * routine, use the IRQ safe variant of the spinlock, see osk_spinlock_irq.
+ * The IRQ safe variant should be used in that situation to prevent
+ * deadlock between a thread/ISR that holds a spinlock while an interrupt occurs
+ * and the interrupt service routine trying to obtain the same spinlock too.
+
+ * The caller must allocate the memory for the @see osk_spinlock
+ * structure, which is then populated within this function. If the OS-specific
+ * spinlock referenced from the structure cannot be initialized, an error is
+ * returned.
+ *
+ * The spinlock must be terminated when no longer required, by using
+ * osk_spinlock_term(). Otherwise, a resource leak may result in the OS.
+ *
+ * The spinlock is initialized with a lock order parameter, \a order. Refer to
+ * @see oskmutex_lockorder for more information on Rwlock/Mutex/Spinlock lock
+ * ordering.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * It is a programming error to attempt to initialize a spinlock that is
+ * currently initialized.
+ *
+ * @param[out] lock pointer to a spinlock structure
+ * @param[in] order the locking order of the spinlock
+ * @return OSK_ERR_NONE on success, any other value indicates a failure.
+ */
+OSK_STATIC_INLINE osk_error osk_spinlock_init(osk_spinlock * const lock, osk_lock_order order) CHECK_RESULT;
+
+/**
+ * @brief Terminate a spinlock
+ *
+ * Terminates the spinlock and releases any associated resources.
+ * The spinlock must be in an unlocked state.
+ *
+ * Terminate the spinlock pointed to by \a lock, which must be
+ * a pointer to a valid unlocked spinlock. When the spinlock is terminated, the
+ * OS-specific spinlock is freed.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * It is a programming error to attempt to terminate a spinlock that is currently
+ * terminated.
+ *
+ * @illegal It is illegal to call osk_spinlock_term() on a locked spinlock.
+ * @param[in] lock pointer to a valid spinlock structure
+ */
+OSK_STATIC_INLINE void osk_spinlock_term(osk_spinlock * lock);
+
+/**
+ * @brief Lock a spinlock
+ *
+ * Lock the spinlock pointed to by \a lock. If the spinlock is currently unlocked,
+ * the calling thread returns with the spinlock locked. If a second thread
+ * attempts to lock the same spinlock, it polls the spinlock until the first thread
+ * unlocks the spinlock. If two or more threads are polling the spinlock waiting
+ * on the first thread to unlock the spinlock, it is undefined as to which thread
+ * will lock the spinlock when the first thread unlocks the spinlock.
+ *
+ * While the spinlock is locked by the calling thread, the spinlock implementation
+ * should prevent any possible deadlock issues arising from another thread on the
+ * same CPU trying to lock the same spinlock.
+ *
+ * While holding a spinlock, you must not sleep. You must not obtain a rwlock,
+ * mutex or do anything else that might block your thread. This is to prevent another
+ * thread trying to lock the same spinlock while your thread holds the spinlock,
+ * which could take a very long time (as it requires your thread to get scheduled
+ * in again and unlock the spinlock) or could even deadlock your system.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * It is a programming error to lock a spinlock, rwlock or mutex with an order that
+ * is higher than any spinlock, rwlock, or mutex held by the current thread. Spinlocks,
+ * Rwlocks, and Mutexes must be locked in the order of highest to lowest, to prevent
+ * deadlocks. Refer to @see oskmutex_lockorder for more information.
+ *
+ * It is a programming error to exit a thread while it has a locked spinlock.
+ *
+ * It is a programming error to lock a spinlock from an ISR context. Use the IRQ
+ * safe spinlock type instead.
+ *
+ * @illegal It is illegal to call osk_spinlock_lock() on a spinlock that is currently
+ * locked by the caller thread. That is, it is illegal for the same thread to
+ * lock a spinlock twice, without unlocking it in between.
+ *
+ * @param[in] lock pointer to a valid spinlock structure
+ */
+OSK_STATIC_INLINE void osk_spinlock_lock(osk_spinlock * lock);
+
+/**
+ * @brief Unlock a spinlock
+ *
+ * Unlock the spinlock pointed to by \a lock. The calling thread must be the
+ * same thread that locked the spinlock. If no other threads are polling the
+ * spinlock waiting on the spinlock to be unlocked, the function returns
+ * immediately, with the spinlock unlocked. If one or more threads are polling
+ * the spinlock waiting on the spinlock to be unlocked, then this function
+ * returns, and a thread waiting on the spinlock can stop polling and continue
+ * with the spinlock locked. It is undefined as to which thread this is.
+ *
+ * @note It is not defined \em when a waiting thread continues. For example,
+ * a thread calling osk_spinlock_unlock() followed by osk_spinlock_lock() may (or may
+ * not) obtain the spinlock again, preventing other threads from continueing.
+ * Neither the 'immediately releasing', nor the 'delayed releasing'
+ * behavior of osk_spinlock_unlock() can be relied upon. If such behavior is
+ * required, then you must implement it yourself, such as by using a second
+ * synchronization primitive.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * @illegal It is illegal for a thread to call osk_spinlock_unlock() on a spinlock
+ * that it has not locked, even if that spinlock is currently locked by another
+ * thread. That is, it is illegal for any thread other than the 'owner' of the
+ * spinlock to unlock it. And, you must not unlock an already unlocked spinlock.
+ *
+ * @param[in] lock pointer to a valid spinlock structure
+ */
+OSK_STATIC_INLINE void osk_spinlock_unlock(osk_spinlock * lock);
+
+/**
+ * @brief Initialize an IRQ safe spinlock
+ *
+ * Initialize an IRQ safe spinlock. If the function returns successfully, the
+ * spinlock is in the unlocked state.
+ *
+ * This variant of spinlock is used for sharing a resource with an interrupt
+ * service routine. The IRQ safe variant should be used in this siutation to
+ * prevent deadlock between a thread/ISR that holds a spinlock while an interrupt
+ * occurs and the interrupt service routine trying to obtain the same spinlock
+ * too. If the spinlock is not used to share a resource with an interrupt service
+ * routine, one should use the osk_spinlock instead of the osk_spinlock_irq
+ * variant, see osk_spinlock_init().
+
+ * The caller must allocate the memory for the @see osk_spinlock_irq
+ * structure, which is then populated within this function. If the OS-specific
+ * spinlock referenced from the structure cannot be initialized, an error is
+ * returned.
+ *
+ * The spinlock must be terminated when no longer required, by using
+ * osk_spinlock_irq_term(). Otherwise, a resource leak may result in the OS.
+ *
+ * The spinlock is initialized with a lock order parameter, \a order. Refer to
+ * @see oskmutex_lockorder for more information on Rwlock/Mutex/Spinlock lock
+ * ordering.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * It is a programming error to attempt to initialize a spinlock that is
+ * currently initialized.
+ *
+ * @param[out] lock pointer to a IRQ safe spinlock structure
+ * @param[in] order the locking order of the IRQ safe spinlock
+ * @return OSK_ERR_NONE on success, any other value indicates a failure.
+ */
+OSK_STATIC_INLINE osk_error osk_spinlock_irq_init(osk_spinlock_irq * const lock, osk_lock_order order) CHECK_RESULT;
+
+/**
+ * @brief Terminate an IRQ safe spinlock
+ *
+ * Terminate the IRQ safe spinlock pointed to by \a lock, which must be
+ * a pointer to a valid unlocked IRQ safe spinlock. When the IRQ safe spinlock
+ * is terminated, the OS-specific spinlock is freed.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * It is a programming error to attempt to terminate a IRQ safe pinlock that is
+ * currently terminated.
+ *
+ * @param[in] lock pointer to a valid IRQ safe spinlock structure
+ */
+OSK_STATIC_INLINE void osk_spinlock_irq_term(osk_spinlock_irq * lock);
+
+/**
+ * @brief Lock an IRQ safe spinlock
+ *
+ * Lock the IRQ safe spinlock (from here on refered to as 'spinlock') pointed to
+ * by \a lock. If the spinlock is currently unlocked, the calling thread returns
+ * with the spinlock locked. If a second thread attempts to lock the same spinlock,
+ * it polls the spinlock until the first thread unlocks the spinlock. If two or
+ * more threads are polling the spinlock waiting on the first thread to unlock the
+ * spinlock, it is undefined as to which thread will lock the spinlock when the
+ * first thread unlocks the spinlock.
+ *
+ * While the spinlock is locked by the calling thread, the spinlock implementation
+ * should prevent any possible deadlock issues arising from another thread on the
+ * same CPU trying to lock the same spinlock.
+ *
+ * While holding a spinlock, you must not sleep. You must not obtain a rwlock,
+ * mutex or do anything else that might block your thread. This is to prevent another
+ * thread trying to lock the same spinlock while your thread holds the spinlock,
+ * which could take a very long time (as it requires your thread to get scheduled
+ * in again and unlock the spinlock) or could even deadlock your system.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * It is a programming error to lock a spinlock, rwlock or mutex with an order that
+ * is higher than any spinlock, rwlock, or mutex held by the current thread. Spinlocks,
+ * Rwlocks, and Mutexes must be locked in the order of highest to lowest, to prevent
+ * deadlocks. Refer to @see oskmutex_lockorder for more information.
+ *
+ * It is a programming error to exit a thread while it has a locked spinlock.
+ *
+ * @illegal It is illegal to call osk_spinlock_irq_lock() on a spinlock that is
+ * currently locked by the caller thread. That is, it is illegal for the same thread
+ * to lock a spinlock twice, without unlocking it in between.
+ *
+ * @param[in] lock pointer to a valid IRQ safe spinlock structure
+ */
+OSK_STATIC_INLINE void osk_spinlock_irq_lock(osk_spinlock_irq * lock);
+
+/**
+ * @brief Unlock an IRQ safe spinlock
+ *
+ * Unlock the IRQ safe spinlock (from hereon refered to as 'spinlock') pointed to
+ * by \a lock. The calling thread/ISR must be the same thread/ISR that locked the
+ * spinlock. If no other threads/ISRs are polling the spinlock waiting on the spinlock
+ * to be unlocked, the function returns* immediately, with the spinlock unlocked. If
+ * one or more threads/ISRs are polling the spinlock waiting on the spinlock to be unlocked,
+ * then this function returns, and a thread/ISR waiting on the spinlock can stop polling
+ * and continue with the spinlock locked. It is undefined as to which thread/ISR this is.
+ *
+ * @note It is not defined \em when a waiting thread/ISR continues. For example,
+ * a thread/ISR calling osk_spinlock_irq_unlock() followed by osk_spinlock_irq_lock() may
+ * (or may not) obtain the spinlock again, preventing other threads from continueing.
+ * Neither the 'immediately releasing', nor the 'delayed releasing'
+ * behavior of osk_spinlock_irq_unlock() can be relied upon. If such behavior is
+ * required, then you must implement it yourself, such as by using a second
+ * synchronization primitive.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * @illegal It is illegal for a thread to call osk_spinlock_irq_unlock() on a spinlock
+ * that it has not locked, even if that spinlock is currently locked by another
+ * thread. That is, it is illegal for any thread other than the 'owner' of the
+ * spinlock to unlock it. And, you must not unlock an already unlocked spinlock.
+ *
+ * @param[in] lock pointer to a valid IRQ safe spinlock structure
+ */
+OSK_STATIC_INLINE void osk_spinlock_irq_unlock(osk_spinlock_irq * lock);
+
+/**
+ * @brief Initialize a rwlock
+ *
+ * Read/write locks allow multiple readers to obtain the lock (shared access),
+ * or one writer to obtain the lock (exclusive access).
+ * Read/write locks are created in an unlocked state.
+ *
+ * Initialize a rwlock structure. If the function returns successfully, the
+ * rwlock is in the unlocked state.
+ *
+ * The caller must allocate the memory for the @see osk_rwlock
+ * structure, which is then populated within this function. If the OS-specific
+ * rwlock referenced from the structure cannot be initialized, an error is
+ * returned.
+ *
+ * The rwlock must be terminated when no longer required, by using
+ * osk_rwlock_term(). Otherwise, a resource leak may result in the OS.
+ *
+ * The rwlock is initialized with a lock order parameter, \a order. Refer to
+ * @see oskmutex_lockorder for more information on Rwlock/Mutex/Spinlock lock
+ * ordering.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * It is a programming error to attempt to initialize a rwlock that is
+ * currently initialized.
+ *
+ * @param[out] lock pointer to a rwlock structure
+ * @param[in] order the locking order of the rwlock
+ * @return OSK_ERR_NONE on success, any other value indicates a failure.
+ */
+OSK_STATIC_INLINE osk_error osk_rwlock_init(osk_rwlock * const lock, osk_lock_order order) CHECK_RESULT;
+
+/**
+ * @brief Terminate a rwlock
+ *
+ * Terminate the rwlock pointed to by \a lock, which must be
+ * a pointer to a valid unlocked rwlock. When the rwlock is terminated, the
+ * OS-specific rwlock is freed.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * It is a programming error to attempt to terminate a rwlock that is currently
+ * terminated.
+ *
+ * @illegal It is illegal to call osk_rwlock_term() on a locked rwlock.
+ *
+ * @param[in] lock pointer to a valid rwlock structure
+ */
+OSK_STATIC_INLINE void osk_rwlock_term(osk_rwlock * lock);
+
+/**
+ * @brief Lock a rwlock for read access
+ *
+ * Lock the rwlock pointed to by \a lock for read access. A rwlock may
+ * be locked for read access by multiple threads. If the mutex
+ * mutex is not locked for exclusive write access, the calling thread
+ * returns with the rwlock locked for read access. If the mutex is
+ * currently locked for exclusive write access, the calling thread blocks
+ * until the thread with exclusive write access unlocks the rwlock.
+ * If multiple threads are blocked waiting for read access or exclusive
+ * write access to the rwlock, it is undefined as to which thread is
+ * unblocked when the rwlock is unlocked (by the thread with exclusive
+ * write access).
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * It is a programming error to lock a rwlock, mutex or spinlock with an order that is
+ * higher than any rwlock, mutex or spinlock held by the current thread. Rwlocks, mutexes and
+ * spinlocks must be locked in the order of highest to lowest, to prevent
+ * deadlocks. Refer to @see oskmutex_lockorder for more information.
+ *
+ * It is a programming error to exit a thread while it has a locked rwlock.
+ *
+ * It is a programming error to lock a rwlock from an ISR context. In an ISR
+ * context you are not allowed to block what osk_rwlock_read_lock() potentially does.
+ *
+ * @illegal It is illegal to call osk_rwlock_read_lock() on a rwlock that is currently
+ * locked by the caller thread. That is, it is illegal for the same thread to
+ * lock a rwlock twice, without unlocking it in between.
+ * @param[in] lock pointer to a valid rwlock structure
+ */
+OSK_STATIC_INLINE void osk_rwlock_read_lock(osk_rwlock * lock);
+
+/**
+ * @brief Unlock a rwlock for read access
+ *
+ * Unlock the rwlock pointed to by \a lock. The calling thread must be the
+ * same thread that locked the rwlock for read access. If no other threads
+ * are waiting on the rwlock to be unlocked, the function returns
+ * immediately, with the rwlock unlocked. If one or more threads are waiting
+ * on the rwlock to be unlocked for write access, and the calling thread
+ * is the last thread holding the rwlock for read access, then this function
+ * returns, and a thread waiting on the rwlock for write access can be
+ * unblocked. It is undefined as to which thread is unblocked.
+ *
+ * @note It is not defined \em when a waiting thread is unblocked. For example,
+ * a thread calling osk_rwlock_read_unlock() followed by osk_rwlock_read_lock()
+ * may (or may not) obtain the lock again, preventing other threads from being
+ * released. Neither the 'immediately releasing', nor the 'delayed releasing'
+ * behavior of osk_rwlock_read_unlock() can be relied upon. If such behavior is
+ * required, then you must implement it yourself, such as by using a second
+ * synchronization primitve.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * @illegal It is illegal for a thread to call osk_rwlock_read_unlock() on a
+ * rwlock that it has not locked, even if that rwlock is currently locked by another
+ * thread. That is, it is illegal for any thread other than the 'owner' of the
+ * rwlock to unlock it. And, you must not unlock an already unlocked rwlock.
+
+ * @param[in] lock pointer to a valid rwlock structure
+ */
+OSK_STATIC_INLINE void osk_rwlock_read_unlock(osk_rwlock * lock);
+
+/**
+ * @brief Lock a rwlock for exclusive write access
+ *
+ * Lock the rwlock pointed to by \a lock for exclusive write access. If the
+ * rwlock is currently unlocked, the calling thread returns with the rwlock
+ * locked. If the rwlock is currently locked, the calling thread blocks
+ * until the last thread with read access or the thread with exclusive write
+ * access unlocks the rwlock. If multiple threads are blocked waiting
+ * for exclusive write access to the rwlock, it is undefined as to which
+ * thread is unblocked when the rwlock is unlocked (by either the last thread
+ * thread with read access or the thread with exclusive write access).
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * It is a programming error to lock a rwlock, mutex or spinlock with an order that is
+ * higher than any rwlock, mutex or spinlock held by the current thread. Rwlocks, mutexes and
+ * spinlocks must be locked in the order of highest to lowest, to prevent
+ * deadlocks. Refer to @see oskmutex_lockorder for more information.
+ *
+ * It is a programming error to exit a thread while it has a locked rwlock.
+ *
+ * It is a programming error to lock a rwlock from an ISR context. In an ISR
+ * context you are not allowed to block what osk_rwlock_write_lock() potentially does.
+ *
+ * @illegal It is illegal to call osk_rwlock_write_lock() on a rwlock that is currently
+ * locked by the caller thread. That is, it is illegal for the same thread to
+ * lock a rwlock twice, without unlocking it in between.
+ *
+ * @param[in] lock pointer to a valid rwlock structure
+ */
+OSK_STATIC_INLINE void osk_rwlock_write_lock(osk_rwlock * lock);
+
+/**
+ * @brief Unlock a rwlock for exclusive write access
+ *
+ * Unlock the rwlock pointed to by \a lock. The calling thread must be the
+ * same thread that locked the rwlock for exclusive write access. If no
+ * other threads are waiting on the rwlock to be unlocked, the function returns
+ * immediately, with the rwlock unlocked. If one or more threads are waiting
+ * on the rwlock to be unlocked, then this function returns, and a thread
+ * waiting on the rwlock can be unblocked. It is undefined as to which
+ * thread is unblocked.
+ *
+ * @note It is not defined \em when a waiting thread is unblocked. For example,
+ * a thread calling osk_rwlock_write_unlock() followed by osk_rwlock_write_lock()
+ * may (or may not) obtain the lock again, preventing other threads from being
+ * released. Neither the 'immediately releasing', nor the 'delayed releasing'
+ * behavior of osk_rwlock_write_unlock() can be relied upon. If such behavior is
+ * required, then you must implement it yourself, such as by using a second
+ * synchronization primitve.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL)
+ * through the \a lock parameter.
+ *
+ * @illegal It is illegal for a thread to call osk_rwlock_write_unlock() on a
+ * rwlock that it has not locked, even if that rwlock is currently locked by another
+ * thread. That is, it is illegal for any thread other than the 'owner' of the
+ * rwlock to unlock it. And, you must not unlock an already unlocked rwlock.
+ *
+ * @param[in] lock pointer to a valid read/write lock structure
+ */
+OSK_STATIC_INLINE void osk_rwlock_write_unlock(osk_rwlock * lock);
+
+/* @} */ /* end group osklocks */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+/* pull in the arch header with the implementation */
+#include <osk/mali_osk_arch_locks.h>
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _OSK_LOCKS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the dedicated memory allocator for the kernel device driver
+ */
+
+#ifndef _OSK_LOW_LEVEL_DEDICATED_MEM_H_
+#define _OSK_LOW_LEVEL_DEDICATED_MEM_H_
+
+#ifdef __KERNEL__
+#include <linux/io.h>
+#endif /* __KERNEL__ */
+
+struct oskp_phy_dedicated_allocator
+{
+ /* lock to protect the free map management */
+ osk_mutex lock;
+
+ osk_phy_addr base;
+ u32 num_pages;
+ u32 free_pages;
+
+ unsigned long * free_map;
+};
+
+OSK_STATIC_INLINE osk_error oskp_phy_dedicated_allocator_init(oskp_phy_dedicated_allocator * const allocator,
+ osk_phy_addr mem, u32 nr_pages, const char* name)
+{
+ osk_error error;
+
+ OSK_ASSERT(allocator);
+ OSK_ASSERT(nr_pages > 0);
+ /* Assert if not page aligned */
+ OSK_ASSERT( 0 == (mem & (OSK_PAGE_SIZE-1)) );
+
+ if (!mem)
+ {
+ /* no address to manage specified */
+ return OSK_ERR_FAIL;
+ }
+ else
+ {
+ u32 i;
+
+ /* try to obtain dedicated memory */
+ if(oskp_phy_dedicated_allocator_request_memory(mem, nr_pages, name) != OSK_ERR_NONE)
+ {
+ /* requested memory not available */
+ return OSK_ERR_FAIL;
+ }
+
+ allocator->base = mem;
+ allocator->num_pages = nr_pages;
+ allocator->free_pages = allocator->num_pages;
+
+ error = osk_mutex_init(&allocator->lock, OSK_LOCK_ORDER_LAST );
+ if (OSK_ERR_NONE != error)
+ {
+ return OSK_ERR_FAIL;
+ }
+
+ allocator->free_map = osk_calloc(sizeof(unsigned long) * ((nr_pages + OSK_BITS_PER_LONG - 1) / OSK_BITS_PER_LONG));
+ if (NULL == allocator->free_map)
+ {
+ osk_mutex_term(&allocator->lock);
+ return OSK_ERR_ALLOC;
+ }
+
+ /* correct for nr_pages not being a multiple of OSK_BITS_PER_LONG */
+ for (i = nr_pages; i < ((nr_pages + OSK_BITS_PER_LONG - 1) & ~(OSK_BITS_PER_LONG-1)); i++)
+ {
+ osk_bitarray_set_bit(i, allocator->free_map);
+ }
+
+ return OSK_ERR_NONE;
+ }
+}
+
+OSK_STATIC_INLINE void oskp_phy_dedicated_allocator_term(oskp_phy_dedicated_allocator *allocator)
+{
+ OSK_ASSERT(allocator);
+ OSK_ASSERT(allocator->free_map);
+ oskp_phy_dedicated_allocator_release_memory(allocator->base, allocator->num_pages);
+ osk_free(allocator->free_map);
+ osk_mutex_term(&allocator->lock);
+}
+
+OSK_STATIC_INLINE u32 oskp_phy_dedicated_pages_alloc(oskp_phy_dedicated_allocator *allocator,
+ u32 nr_pages, osk_phy_addr *pages)
+{
+ u32 pages_allocated;
+
+ OSK_ASSERT(pages);
+ OSK_ASSERT(allocator);
+ OSK_ASSERT(allocator->free_map);
+
+ osk_mutex_lock(&allocator->lock);
+
+ for (pages_allocated = 0; pages_allocated < OSK_MIN(nr_pages, allocator->free_pages); pages_allocated++)
+ {
+ u32 pfn;
+ void * mapping;
+
+ pfn = osk_bitarray_find_first_zero_bit(allocator->free_map, allocator->num_pages);
+ /* As the free_pages test passed ffz should never fail */
+ OSK_ASSERT(pfn != allocator->num_pages);
+
+ /* mark as allocated */
+ osk_bitarray_set_bit(pfn, allocator->free_map);
+
+ /* find phys addr of the page */
+ pages[pages_allocated] = allocator->base + (pfn << OSK_PAGE_SHIFT);
+
+#ifdef __KERNEL__
+ /* zero the page */
+ if(OSK_SIMULATE_FAILURE(OSK_OSK))
+ {
+ mapping = NULL;
+ }
+ else
+ {
+ mapping = ioremap_wc(pages[pages_allocated], SZ_4K);
+ }
+#else
+ mapping = osk_kmap(pages[pages_allocated]);
+#endif /* __KERNEL__ */
+
+ if (NULL == mapping)
+ {
+ /* roll back */
+ for (pages_allocated++; pages_allocated > 0; pages_allocated--)
+ {
+ pfn = (pages[pages_allocated-1] - allocator->base) >> OSK_PAGE_SHIFT;
+ osk_bitarray_clear_bit(pfn, allocator->free_map);
+ }
+ break;
+ }
+
+ OSK_MEMSET(mapping, 0x00, OSK_PAGE_SIZE);
+
+ osk_sync_to_memory(pages[pages_allocated], mapping, OSK_PAGE_SIZE);
+#ifdef __KERNEL__
+ iounmap(mapping);
+#else
+ osk_kunmap(pages[pages_allocated], mapping);
+#endif /* __KERNEL__ */
+ }
+
+ allocator->free_pages -= pages_allocated;
+ osk_mutex_unlock(&allocator->lock);
+
+ return pages_allocated;
+}
+
+OSK_STATIC_INLINE void oskp_phy_dedicated_pages_free(oskp_phy_dedicated_allocator *allocator,
+ u32 nr_pages, osk_phy_addr *pages)
+{
+ u32 i;
+
+ OSK_ASSERT(pages);
+ OSK_ASSERT(allocator);
+ OSK_ASSERT(allocator->free_map);
+
+ osk_mutex_lock(&allocator->lock);
+
+ for (i = 0; i < nr_pages; i++)
+ {
+ if (0 != pages[i])
+ {
+ u32 pfn;
+
+ OSK_ASSERT(pages[i] >= allocator->base);
+ OSK_ASSERT(pages[i] < allocator->base + (allocator->num_pages << OSK_PAGE_SHIFT));
+
+ pfn = (pages[i] - allocator->base) >> OSK_PAGE_SHIFT;
+ osk_bitarray_clear_bit(pfn, allocator->free_map);
+
+ allocator->free_pages++;
+
+ pages[i] = 0;
+ }
+ }
+
+ osk_mutex_unlock(&allocator->lock);
+}
+
+#endif /* _OSK_LOW_LEVEL_DEDICATED_MEM_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_low_level_mem.h
+ *
+ * Defines the kernel low level memory abstraction layer for the base
+ * driver.
+ */
+
+#ifndef _OSK_LOW_LEVEL_MEM_H_
+#define _OSK_LOW_LEVEL_MEM_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @addtogroup osklowlevelmem Low level memory
+ *
+ * Provides functions to allocate physical memory and ensure cache coherency.
+ *
+ * @{
+ */
+
+/**
+ * CPU virtual address
+ */
+typedef void *osk_virt_addr;
+
+/**
+ * Physical page allocator
+ */
+typedef struct osk_phy_allocator osk_phy_allocator;
+
+/**
+ * Dedicated physical page allocator
+ */
+typedef struct oskp_phy_os_allocator oskp_phy_os_allocator;
+/**
+ * OS physical page allocator
+ */
+typedef struct oskp_phy_dedicated_allocator oskp_phy_dedicated_allocator;
+
+/**
+ * @brief Initialize a physical page allocator
+ *
+ * The physical page allocator is responsible for allocating physical memory pages of
+ * OSK_PAGE_SIZE bytes each. Pages are allocated through the OS or from a reserved
+ * memory region.
+ *
+ * Physical page allocation through the OS
+ *
+ * If \a mem is 0, upto \a nr_pages of pages may be allocated through the OS for use
+ * by a user process. OSs that require allocating CPU virtual address space in order
+ * to allocate physical pages must observe that the CPU virtual address space is
+ * allocated for the current user process and that the physical allocator must always
+ * be used with this same user process.
+ *
+ * If \a mem is 0, and \a nr_pages is 0, a variable number of pages may be allocated
+ * through the OS for use by the kernel (only limited by the available OS memory).
+ * Allocated pages may be mapped into the kernel using osk_kmap(). The use case for
+ * this type of physical allocator is the allocation of physical pages for MMU page
+ * tables. OSs that require allocating CPU virtual address space in order
+ * to allocate physical pages must likely manage a list of fixed size virtual
+ * address regions against which pages are committed as more pages are allocated.
+ *
+ * Physical page allocation from a reserved memory region
+ *
+ * If \a mem is not 0, \a mem specifies the physical start address of a physically
+ * contiguous memory region, from which \a nr_pages of pages may be allocated, for
+ * use by a user process. The start address is aligned to OSK_PAGE_SIZE bytes.
+ * The memory region must not be in use by the OS and solely for use by the physical
+ * allocator. OSs that require allocating CPU virtual address space in order
+ * to allocate physical pages must observe that the CPU virtual address space is
+ * allocated for the current user process and that the physical allocator must always
+ * be used with this same user process.
+ *
+ * @param[out] allocator physical allocator to initialize
+ * @param[in] mem Set \a mem to 0 if physical pages should be allocated through the OS,
+ * otherwise \a mem represents the physical address of a reserved
+ * memory region from which pages should be allocated. The physical
+ * address must be OSK_PAGE_SIZE aligned.
+ * @param[in] nr_pages maximum number of physical pages that can be allocated.
+ * If nr_pages > 0, pages are for use in user space.
+ * If nr_pages is 0, a variable number number of pages can be allocated
+ * (limited by the available pages from the OS) but the pages are
+ * for use by the kernel and \a mem must be set to 0
+ * (to enable allocating physical pages through the OS).
+ * @param[in] name name of the reserved memory region
+ * @return OSK_ERR_NONE if successful. Any other value indicates failure.
+ */
+OSK_STATIC_INLINE osk_error osk_phy_allocator_init(osk_phy_allocator * const allocator, osk_phy_addr mem, u32 nr_pages, const char* name) CHECK_RESULT;
+
+OSK_STATIC_INLINE osk_error oskp_phy_os_allocator_init(oskp_phy_os_allocator * const allocator,
+ osk_phy_addr mem, u32 nr_pages) CHECK_RESULT;
+OSK_STATIC_INLINE osk_error oskp_phy_dedicated_allocator_init(oskp_phy_dedicated_allocator * const allocator,
+ osk_phy_addr mem, u32 nr_pages, const char* name) CHECK_RESULT;
+OSK_STATIC_INLINE osk_error oskp_phy_dedicated_allocator_request_memory(osk_phy_addr mem,u32 nr_pages, const char* name) CHECK_RESULT;
+
+
+/**
+ * @brief Terminate a physical page allocator
+ *
+ * Frees any resources necessary to manage the physical allocator. Any physical pages that
+ * were allocated or mapped by the allocator must have been freed and unmapped earlier.
+ *
+ * Allocating and mapping pages using the terminated allocator is prohibited until the
+ * the \a allocator is reinitailized with osk_phy_allocator_init().
+ *
+ * @param[in] allocator initialized physical allocator
+ */
+OSK_STATIC_INLINE void osk_phy_allocator_term(osk_phy_allocator *allocator);
+
+OSK_STATIC_INLINE void oskp_phy_os_allocator_term(oskp_phy_os_allocator *allocator);
+OSK_STATIC_INLINE void oskp_phy_dedicated_allocator_term(oskp_phy_dedicated_allocator *allocator);
+OSK_STATIC_INLINE void oskp_phy_dedicated_allocator_release_memory(osk_phy_addr mem,u32 nr_pages);
+
+/**
+ * @brief Allocate physical pages
+ *
+ * Allocates \a nr_pages physical pages of OSK_PAGE_SIZE each using the physical
+ * allocator \a allocator and stores the physical address of each allocated page
+ * in the \a pages array.
+ *
+ * If the physical allocator was initialized to allocate pages for use by a user
+ * process, the pages need to be allocated in the same user space context as the
+ * physical allocator was initialized in.
+ *
+ * This function may block and cannot be used from ISR context.
+ *
+ * @param[in] allocator initialized physical allocator
+ * @param[in] nr_pages number of physical pages to allocate
+ * @param[out] pages array of \a nr_pages elements storing the physical
+ * address of an allocated page
+ * @return The number of pages successfully allocated,
+ * which might be lower than requested, including zero pages.
+ */
+OSK_STATIC_INLINE u32 osk_phy_pages_alloc(osk_phy_allocator *allocator, u32 nr_pages, osk_phy_addr *pages) CHECK_RESULT;
+
+OSK_STATIC_INLINE u32 oskp_phy_os_pages_alloc(oskp_phy_os_allocator *allocator,
+ u32 nr_pages, osk_phy_addr *pages) CHECK_RESULT;
+OSK_STATIC_INLINE u32 oskp_phy_dedicated_pages_alloc(oskp_phy_dedicated_allocator *allocator,
+ u32 nr_pages, osk_phy_addr *pages) CHECK_RESULT;
+
+/**
+ * @brief Free physical pages
+ *
+ * Frees physical pages previously allocated by osk_phy_pages_alloc(). The same
+ * arguments used for the allocation need to be specified when freeing them.
+ *
+ * Freeing individual pages of a set of pages allocated by osk_phy_pages_alloc()
+ * is not allowed.
+ *
+ * If the physical allocator was initialized to allocate pages for use by a user
+ * process, the pages need to be freed in the same user space context as the
+ * physical allocator was initialized in.
+ *
+ * The contents of the \a pages array is undefined after osk_phy_pages_free has
+ * freed the pages.
+ *
+ * @param[in] allocator initialized physical allocator
+ * @param[in] nr_pages number of physical pages to free (as used during the allocation)
+ * @param[in] pages array of \a nr_pages storing the physical address of an
+ * allocated page (as used during the allocation).
+ */
+OSK_STATIC_INLINE void osk_phy_pages_free(osk_phy_allocator *allocator, u32 nr_pages, osk_phy_addr *pages);
+
+OSK_STATIC_INLINE void oskp_phy_os_pages_free(oskp_phy_os_allocator *allocator,
+ u32 nr_pages, osk_phy_addr *pages);
+OSK_STATIC_INLINE void oskp_phy_dedicated_pages_free(oskp_phy_dedicated_allocator *allocator,
+ u32 nr_pages, osk_phy_addr *pages);
+/**
+ * @brief Map a physical page into the kernel
+ *
+ * Maps a physical page that was previously allocated by osk_phy_pages_alloc()
+ * with a physical allocator setup for allocating pages for use by the kernel,
+ * @see osk_phy_allocator_init().
+ *
+ * Notes:
+ * - Kernel virtual memory is limited. Limit the number of pages mapped into
+ * the kernel and limit the duration of the mapping.
+ *
+ * @param[in] page physical address of the page to unmap
+ * @return CPU virtual address in the kernel, NULL in case of a failure.
+ */
+OSK_STATIC_INLINE void *osk_kmap(osk_phy_addr page) CHECK_RESULT;
+
+/**
+ * @brief Unmap a physical page from the kernel
+ *
+ * Unmaps a previously mapped physical page (with osk_kmap) from the kernel.
+ *
+ * @param[in] page physical address of the page to unmap
+ * @param[in] mapping virtual address of the mapping to unmap
+ */
+OSK_STATIC_INLINE void osk_kunmap(osk_phy_addr page, void * mapping);
+
+/**
+ * @brief Map a physical page into the kernel
+ *
+ * Maps a physical page that was previously allocated by osk_phy_pages_alloc()
+ * with a physical allocator setup for allocating pages for use by the kernel,
+ * @see osk_phy_allocator_init().
+ *
+ * Notes:
+ * @li Used for mapping a single page for a very short duration
+ * @li The system only supports limited number of atomic mappings,
+ * so use should be limited
+ * @li The caller must not sleep until after the osk_kunmap_atomic is called.
+ * @li It may be assumed that osk_k[un]map_atomic will not fail.
+ *
+ * @param[in] page physical address of the page to unmap
+ * @return CPU virtual address in the kernel, NULL in case of a failure.
+ */
+OSK_STATIC_INLINE void *osk_kmap_atomic(osk_phy_addr page) CHECK_RESULT;
+
+/**
+ * @brief Unmap a physical page from the kernel
+ *
+ * Unmaps a previously mapped physical page (with osk_kmap_atomic) from the kernel.
+ *
+ * @param[in] page physical address of the page to unmap
+ * @param[in] mapping virtual address of the mapping to unmap
+ */
+OSK_STATIC_INLINE void osk_kunmap_atomic(osk_phy_addr page, void * mapping);
+
+/**
+ * A pointer to a cache synchronization function, either osk_sync_to_cpu()
+ * or osk_sync_to_memory().
+ */
+typedef void (*osk_sync_kmem_fn)(osk_phy_addr, osk_virt_addr, size_t);
+
+/**
+ * @brief Synchronize a memory area for other system components usage
+ *
+ * Performs the necessary memory coherency operations on a given memory area,
+ * such that after the call, changes in memory are correctly seen by other
+ * system components. Any change made to memory after that call may not be seen
+ * by other system components.
+ *
+ * In effect:
+ * - all CPUs will perform a cache clean operation on their inner & outer data caches
+ * - any write buffers are drained (including that of outer cache controllers)
+ *
+ * This function waits until all operations have completed.
+ *
+ * The area is restricted to one page or less and must not cross a page boundary.
+ * The offset within the page is aligned to cache line size and size is ensured
+ * to be a multiple of the cache line size.
+ *
+ * Both physical and virtual address of the area need to be provided to support OS
+ * cache flushing APIs that either use the virtual or the physical address. When
+ * called from OS specific code it is allowed to only provide the address that
+ * is actually used by the specific OS and leave the other address as 0.
+ *
+ * @param[in] paddr physical address
+ * @param[in] vaddr CPU virtual address valid in the current user VM or the kernel VM
+ * @param[in] sz size of the area, <= OSK_PAGE_SIZE.
+ */
+OSK_STATIC_INLINE void osk_sync_to_memory(osk_phy_addr paddr, osk_virt_addr vaddr, size_t sz);
+
+/**
+ * @brief Synchronize a memory area for CPU usage
+ *
+ * Performs the necessary memory coherency operations on a given memory area,
+ * such that after the call, changes in memory are correctly seen by any CPU.
+ * Any change made to this area by any CPU before this call may be lost.
+ *
+ * In effect:
+ * - all CPUs will perform a cache clean & invalidate operation on their inner &
+ * outer data caches.
+ *
+ * @note Stricly only an invalidate operation is required but by cleaning the cache
+ * too we prevent loosing changes made to the memory area due to software bugs. By
+ * having these changes cleaned from the cache it allows us to catch the memory
+ * area getting corrupted with the help of watch points. In correct operation the
+ * clean & invalidate operation would not be more expensive than an invalidate
+ * operation. Also note that for security reasons, it is dangerous to expose a
+ * cache 'invalidate only' operation to user space.
+ *
+ * - any read buffers are flushed (including that of outer cache controllers)
+ *
+ * This function waits until all operations have completed.
+ *
+ * The area is restricted to one page or less and must not cross a page boundary.
+ * The offset within the page is aligned to cache line size and size is ensured
+ * to be a multiple of the cache line size.
+ *
+ * Both physical and virtual address of the area need to be provided to support OS
+ * cache flushing APIs that either use the virtual or the physical address. When
+ * called from OS specific code it is allowed to only provide the address that
+ * is actually used by the specific OS and leave the other address as 0.
+ *
+ * @param[in] paddr physical address
+ * @param[in] vaddr CPU virtual address valid in the current user VM or the kernel VM
+ * @param[in] sz size of the area, <= OSK_PAGE_SIZE.
+ */
+OSK_STATIC_INLINE void osk_sync_to_cpu(osk_phy_addr paddr, osk_virt_addr vaddr, size_t sz);
+
+/** @} */ /* end group osklowlevelmem */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+/* pull in the arch header with the implementation */
+#include "mali_osk_low_level_dedicated_mem.h"
+#include <osk/mali_osk_arch_low_level_mem.h>
+
+typedef enum oskp_phy_allocator_type
+{
+ OSKP_PHY_ALLOCATOR_OS,
+ OSKP_PHY_ALLOCATOR_DEDICATED
+} oskp_phy_allocator_type;
+
+struct osk_phy_allocator
+{
+ oskp_phy_allocator_type type;
+ union {
+ struct oskp_phy_dedicated_allocator dedicated;
+ struct oskp_phy_os_allocator os;
+ } data;
+};
+
+
+OSK_STATIC_INLINE osk_error osk_phy_allocator_init(osk_phy_allocator * const allocator, osk_phy_addr mem, u32 nr_pages, const char* name)
+{
+ OSK_ASSERT(allocator);
+ if (mem == 0)
+ {
+ allocator->type = OSKP_PHY_ALLOCATOR_OS;
+ return oskp_phy_os_allocator_init(&allocator->data.os, mem, nr_pages);
+ }
+ else
+ {
+ allocator->type = OSKP_PHY_ALLOCATOR_DEDICATED;
+ return oskp_phy_dedicated_allocator_init(&allocator->data.dedicated, mem, nr_pages, name);
+ }
+}
+
+OSK_STATIC_INLINE void osk_phy_allocator_term(osk_phy_allocator *allocator)
+{
+ OSK_ASSERT(allocator);
+ if (allocator->type == OSKP_PHY_ALLOCATOR_OS)
+ {
+ oskp_phy_os_allocator_term(&allocator->data.os);
+ }
+ else
+ {
+ oskp_phy_dedicated_allocator_term(&allocator->data.dedicated);
+ }
+}
+
+OSK_STATIC_INLINE u32 osk_phy_pages_alloc(osk_phy_allocator *allocator, u32 nr_pages, osk_phy_addr *pages)
+{
+ OSK_ASSERT(allocator);
+ OSK_ASSERT(pages);
+ if (allocator->type != OSKP_PHY_ALLOCATOR_OS && allocator->type != OSKP_PHY_ALLOCATOR_DEDICATED)
+ {
+ return 0;
+ }
+ if (allocator->type == OSKP_PHY_ALLOCATOR_OS)
+ {
+ return oskp_phy_os_pages_alloc(&allocator->data.os, nr_pages, pages);
+ }
+ else
+ {
+ return oskp_phy_dedicated_pages_alloc(&allocator->data.dedicated, nr_pages, pages);
+ }
+}
+
+OSK_STATIC_INLINE void osk_phy_pages_free(osk_phy_allocator *allocator, u32 nr_pages, osk_phy_addr *pages)
+{
+ OSK_ASSERT(allocator);
+ OSK_ASSERT(pages);
+ if (allocator->type == OSKP_PHY_ALLOCATOR_OS)
+ {
+ oskp_phy_os_pages_free(&allocator->data.os, nr_pages, pages);
+ }
+ else
+ {
+ oskp_phy_dedicated_pages_free(&allocator->data.dedicated, nr_pages, pages);
+ }
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _OSK_LOW_LEVEL_MEM_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_math.h
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_MATH_H_
+#define _OSK_MATH_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @addtogroup oskmath Math
+ *
+ * Math related functions for which no commmon behavior exists on OS.
+ *
+ * @{
+ */
+
+/**
+ * @brief Divide a 64-bit value with a 32-bit divider
+ *
+ * Performs an (unsigned) integer division of a 64-bit value
+ * with a 32-bit divider and returns the 64-bit result and
+ * 32-bit remainder.
+ *
+ * Provided as part of the OSK as not all OSs support 64-bit
+ * division in an uniform way. Currently required to support
+ * printing 64-bit numbers in the OSK debug message functions.
+ *
+ * @param[in,out] value pointer to a 64-bit value to be divided by
+ * \a divisor. The integer result of the division
+ * is stored in \a value on output.
+ * @param[in] divisor 32-bit divisor
+ * @return 32-bit remainder of the division
+ */
+OSK_STATIC_INLINE u32 osk_divmod6432(u64 *value, u32 divisor);
+
+/** @} */ /* end group oskmath */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+/* pull in the arch header with the implementation */
+#include <osk/mali_osk_arch_math.h>
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _OSK_MATH_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_mem.h
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_MEM_H_
+#define _OSK_MEM_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+/* pull in the arch header with the implementation */
+#include <osk/mali_osk_arch_mem.h>
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @addtogroup oskmem Memory
+ *
+ * Provides C standard library style memory allocation functions (e.g. malloc, free).
+ *
+ * @{
+ */
+
+/**
+ * @brief Allocate kernel heap memory
+ *
+ * Returns a buffer capable of containing at least \a size bytes. The
+ * contents of the buffer are undefined.
+ *
+ * The buffer is suitably aligned for storage and subsequent access of every
+ * type that the compiler supports. Therefore, the pointer to the start of the
+ * buffer may be cast into any pointer type, and be subsequently accessed from
+ * such a pointer, without loss of information.
+ *
+ * When the buffer is no longer in use, it must be freed with osk_free().
+ * Failure to do so will cause a memory leak.
+ *
+ * @note For the implementor: most toolchains supply memory allocation
+ * functions that meet the compiler's alignment requirements. Therefore, there
+ * is often no need to write code to align the pointer returned by your
+ * system's memory allocator. Refer to your system's memory allocator for more
+ * information (e.g. the malloc() function, if used).
+ *
+ * The buffer can be accessed by all threads in the kernel. You need
+ * not free the buffer from the same thread that allocated the memory.
+ *
+ * May block while allocating memory and is therefore not allowed in interrupt
+ * service routines.
+ *
+ * @illegal It is illegal to call osk_malloc() with \a size == 0.
+ *
+ * @param size Number of bytes to allocate
+ * @return On success, the buffer allocated. NULL on failure.
+ *
+ */
+OSK_STATIC_INLINE void *osk_malloc(size_t size) CHECK_RESULT;
+
+/**
+ * @brief Allocate and zero kernel heap memory
+ *
+ * Returns a buffer capable of containing at least \a size bytes.
+ * The buffer is initialized to zero.
+ *
+ * The buffer is suitably aligned for storage and subsequent access of every
+ * type that the compiler supports. Therefore, the pointer to the start of the
+ * buffer may be cast into any pointer type, and be subsequently accessed from
+ * such a pointer, without loss of information.
+ *
+ * When the buffer is no longer in use, it must be freed with osk_free().
+ * Failure to do so will cause a memory leak.
+ *
+ * @note For the implementor: most toolchains supply memory allocation
+ * functions that meet the compiler's alignment requirements. Therefore, there
+ * is often no need to write code to align the pointer returned by your
+ * system's memory allocator. Refer to your system's memory allocator for more
+ * information (e.g. the malloc() function, if used).
+ *
+ * The buffer can be accessed by all threads in the kernel. You need
+ * not free the buffer from the same thread that allocated the memory.
+ *
+ * May block while allocating memory and is therefore not allowed in interrupt
+ * service routines.
+ *
+ * @illegal It is illegal to call osk_calloc() with \a size == 0.
+ *
+ * @param[in] size number of bytes to allocate
+ * @return On success, the zero initialized buffer allocated. NULL on failure
+ */
+OSK_STATIC_INLINE void *osk_calloc(size_t size);
+
+/**
+ * @brief Free kernel heap memory
+ *
+ * Reclaims the buffer pointed to by the parameter \a ptr for the kernel.
+ * All memory returned from osk_malloc() and osk_calloc() must
+ * be freed before the kernel driver exits. Otherwise, a memory leak will
+ * occur.
+ *
+ * Memory must be freed once. It is an error to free the same non-NULL pointer
+ * more than once.
+ *
+ * It is legal to free the NULL pointer.
+ *
+ * @param[in] ptr Pointer to buffer to free
+ *
+ * May block while freeing memory and is therefore not allowed in interrupt
+ * service routines.
+ *
+ * @param[in] ptr pointer to memory previously allocated by
+ * osk_malloc() or osk_calloc(). If ptr is NULL no operation
+ * is performed.
+ */
+OSK_STATIC_INLINE void osk_free(void * ptr);
+
+/**
+ * @brief Allocate kernel memory at page granularity suitable for mapping into user space
+ *
+ * Allocates a number of pages from kernel virtual memory to store at least \a size bytes.
+ *
+ * The allocated memory is aligned to OSK_PAGE_SIZE bytes and is allowed to be mapped
+ * into user space with read/write access.
+ *
+ * The allocated memory is initialized to zero to prevent any data leaking from kernel space.
+ * One needs to be aware not to store any kernel objects or pointers here as these
+ * could be modified by the user at any time.
+ *
+ * If \a size is not a multiple of OSK_PAGE_SIZE, the last page of the allocation is
+ * only partially used. It is not allowed to store any data in the unused area of the
+ * last page.
+ *
+ * @illegal It is illegal to call osk_vmalloc() with \a size == 0.
+
+ * May block while allocating memory and is therefore not allowed in interrupt
+ * service routines.
+ *
+ * @param[in] size number of bytes to allocate (will be rounded up to
+ * a multiple of OSK_PAGE_SIZE).
+ * @return pointer to allocated memory, NULL on failure
+ */
+OSK_STATIC_INLINE void *osk_vmalloc(size_t size) CHECK_RESULT;
+
+/**
+ * @brief Free kernel memory
+ *
+ * Releases memory to the kernel, previously allocated with osk_vmalloc().
+ * The same pointer returned from osk_vmalloc() needs to be provided --
+ * freeing portions of an allocation is not allowed.
+ *
+ * May block while freeing memory and is therefore not allowed in interrupt
+ * service routines.
+ *
+ * @param[in] vaddr pointer to memory previously allocated by osk_vmalloc().
+ * If vaddr is NULL no operation is performed.
+ */
+OSK_STATIC_INLINE void osk_vfree(void *vaddr);
+
+#ifndef OSK_MEMSET
+/** @brief Fills memory.
+ *
+ * Sets the first \a size bytes of the block of memory pointed to by \a ptr to
+ * the specified value
+ * @param[out] ptr Pointer to the block of memory to fill.
+ * @param[in] chr Value to be set, passed as an int. The byte written into
+ * memory will be the smallest positive integer equal to (\a
+ * chr mod 256).
+ * @param[in] size Number of bytes to be set to the value.
+ * @return \a ptr is always passed through unmodified
+ *
+ * @note the prototype of the function is:
+ * @code void *OSK_MEMSET( void *ptr, int chr, size_t size ); @endcode
+ */
+#define OSK_MEMSET( ptr, chr, size ) You_must_define_the_OSK_MEMSET_macro_in_the_platform_layer
+
+#error You must define the OSK_MEMSET macro in the mali_osk_arch_mem.h layer.
+
+/* The definition was only provided for documentation purposes; remove it now. */
+#undef OSK_MEMSET
+#endif /* OSK_MEMSET */
+
+#ifndef OSK_MEMCPY
+/** @brief Copies memory.
+ *
+ * Copies the \a len bytes from the buffer pointed by the parameter \a src
+ * directly to the buffer pointed by \a dst.
+ *
+ * @illegal It is illegal to call OSK_MEMCPY with \a src overlapping \a
+ * dst anywhere in \a len bytes.
+ *
+ * @param[out] dst Pointer to the destination array where the content is to be copied.
+ * @param[in] src Pointer to the source of data to be copied.
+ * @param[in] len Number of bytes to copy.
+ * @return \a dst is always passed through unmodified.
+ *
+ * @note the prototype of the function is:
+ * @code void *OSK_MEMCPY( void *dst, CONST void *src, size_t len ); @endcode
+ */
+#define OSK_MEMCPY( dst, src, len ) You_must_define_the_OSK_MEMCPY_macro_in_the_platform_layer
+
+#error You must define the OSK_MEMCPY macro in the mali_osk_arch_mem.h file.
+
+/* The definition was only provided for documentation purposes; remove it now. */
+#undef OSK_MEMCPY
+#endif /* OSK_MEMCPY */
+
+#ifndef OSK_MEMCMP
+/** @brief Compare memory areas
+ *
+ * Compares \a len bytes of the memory areas pointed by the parameter \a s1 and
+ * \a s2.
+ *
+ * @param[in] s1 Pointer to the first area of memory to compare.
+ * @param[in] s2 Pointer to the second area of memory to compare.
+ * @param[in] len Number of bytes to compare.
+ * @return an integer less than, equal to, or greater than zero if the first
+ * \a len bytes of s1 is found, respectively, to be less than, to match, or
+ * be greater than the first \a len bytes of s2.
+ *
+ * @note the prototype of the function is:
+ * @code int OSK_MEMCMP( CONST void *s1, CONST void *s2, size_t len ); @endcode
+ */
+#define OSK_MEMCMP( s1, s2, len ) You_must_define_the_OSK_MEMCMP_macro_in_the_platform_layer
+
+#error You must define the OSK_MEMCMP macro in the mali_osk_arch_mem.h file.
+
+/* The definition was only provided for documentation purposes; remove it now. */
+#undef OSK_MEMCMP
+#endif /* OSK_MEMCMP */
+
+/** @} */ /* end group oskmem */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+/* Include osk alloc wrappers header */
+
+#if (1 == MALI_BASE_TRACK_MEMLEAK)
+#include "osk/include/mali_osk_mem_wrappers.h"
+#endif
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _OSK_MEM_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _OSK_POWER_H_
+#define _OSK_POWER_H_
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @addtogroup oskpower Power management
+ *
+ * Some OS need to approve power state changes to a device, either to
+ * control power to a bus the device resides on, or to give the OS
+ * power manager the chance to see if the power state change is allowed
+ * for the current OS power management policy.
+ *
+ * @{
+ */
+
+/**
+ * @brief Request and perform a change in the power state of the device
+ *
+ * A function to request the OS to perform a change the power state of a device. This
+ * function returns when the power state change has completed.
+ *
+ * This allows the OS to control the power for the bus on which the GPU device resides,
+ * and the OS power manager can verify changing the power state is allowed according to
+ * its own power management policy (the OS may have been informed that an application will
+ * make heavy use of the GPU soon). As a result of the request the OS is likely to
+ * request the GPU device driver to actually perform the power state change (in Windows
+ * CE for instance, the OS power manager will issue an IOCTL_POWER_SET to actually make
+ * the GPU device change the power state).
+ *
+ * The result of the request is either success (the GPU device driver has successfully
+ * completed the power state change for the GPU device), refused (the OS didn't allow
+ * the power state change), or failure (the GPU device driver encountered an error
+ * changing the power state).
+ *
+ * @param[in,out] info OS specific information necessary to control power to the device
+ * @param[in] state power state to switch to (off, idle, or active)
+ * @return OSK_POWER_REQUEST_FINISHED when the driver successfully completed the power
+ * state change for the device, OSK_POWER_REQUEST_FAILED when it failed, or
+ * OSK_POWER_REQUEST_REFUSED when the OS didn't allow the power state change.
+ */
+OSK_STATIC_INLINE osk_power_request_result osk_power_request(osk_power_info *info, osk_power_state state) CHECK_RESULT;
+
+/* @} */ /* end group oskpower */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+/* pull in the arch header with the implementation */
+#include <osk/mali_osk_arch_power.h>
+
+#endif /* _OSK_POWER_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_time.h
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_TIME_H_
+#define _OSK_TIME_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @addtogroup osktime Time
+ *
+ * A set of time functions based on the OS tick timer which allow
+ * - retrieving the current tick count
+ * - determining if a deadline has passed
+ * - calculating the elapsed time between two tick counts
+ * - converting a time interval to a tick count
+ *
+ * Used in combination with scheduling timers, or determining duration of certain actions (timeouts, profiling).
+ * @{
+ */
+
+/**
+ * @brief Get current tick count
+ *
+ * Retrieves the current tick count in the OS native resolution. Use in
+ * combination with osk_time_after() to determine if current time has
+ * passed a deadline, or to determine the elapsed time with osk_time_tickstoms()
+ * and osk_time_after().
+ *
+ * The tick count resolution is 32-bit or higher, supporting a 1000Hz tick count
+ * of 2^32/1000*60*60*24 ~= 49.7 days before the counter wraps.
+ *
+ * @return current tick count
+ */
+OSK_STATIC_INLINE osk_ticks osk_time_now(void) CHECK_RESULT;
+
+/**
+ * @brief Convert milliseconds to ticks
+ *
+ * Converts \a ms milliseconds to ticks.
+ *
+ * Intended use of this function is to convert small timeout periods or
+ * calculate nearby deadlines (e.g. osk_ticks_now() + osk_time_mstoticks(50)).
+ *
+ * Supports converting a period of up to 2^32/1000*60*60*24 ~= 49.7 days
+ * in the case of a 1000Hz OS tick timer.
+ *
+ * @param[in] ms number of millisecons to convert to ticks
+ * @return number of ticks in \a ms milliseconds
+ */
+OSK_STATIC_INLINE u32 osk_time_mstoticks(u32 ms) CHECK_RESULT;
+
+/**
+ * @brief Calculate elapsed time
+ *
+ * Calculates elapsed time in milliseconds between tick \a ticka and \a tickb,
+ * taking into account that the tick counter may have wrapped. Note that
+ * \a tickb must be later than \a ticka in time.
+ *
+ * @param[in] ticka a tick count value (as returned from osk_time_now())
+ * @param[in] tickb a tick count value (as returned from osk_time_now())
+ * @return elapsed time in milliseconds
+ */
+OSK_STATIC_INLINE u32 osk_time_elapsed(osk_ticks ticka, osk_ticks tickb) CHECK_RESULT;
+
+/**
+ * @brief Determines which tick count is later in time
+ *
+ * Determines if \a ticka comes after \a tickb in time. Handles the case where
+ * the tick counter may have wrapped.
+ *
+ * Intended use of this function is to determine if a deadline has passed
+ * or to determine how the difference between two tick count values should
+ * be calculated.
+ *
+ * @param[in] ticka a tick count value (as returned from osk_time_now())
+ * @param[in] tickb a tick count value (as returned from osk_time_now())
+ * @return MALI_TRUE when \a ticka is after \a tickb in time.
+ * @return MALI_FALSE when \a tickb is after \a ticka in time.
+ */
+OSK_STATIC_INLINE mali_bool osk_time_after(osk_ticks ticka, osk_ticks tickb) CHECK_RESULT;
+
+/**
+ * @brief Retrieve current "wall clock" time
+ *
+ * This function returns the current time in a format that userspace can also
+ * produce and allows direct comparison of events in the kernel with events
+ * that userspace controls.
+ *
+ * @param[in] ts An osk_timespec structure
+ */
+OSK_STATIC_INLINE void osk_gettimeofday(osk_timeval *ts);
+
+/* @} */ /* end group osktime */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+/* pull in the arch header with the implementation */
+#include <osk/mali_osk_arch_time.h>
+
+#ifdef __cplusplus
+}
+#endif
+
+
+#endif /* _OSK_TIME_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_timers.h
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_TIMERS_H_
+#define _OSK_TIMERS_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @addtogroup osktimers Timers
+ *
+ * A set of functions to control a one-shot timer. Timer expirations are
+ * set relative to current time at millisecond resolution. A minimum time
+ * period of 1 millisecond needs to be observed -- immediate firing of
+ * timers is not supported. A user-supplied function is called with a
+ * user-supplied argument when the timer expires. The expiration time
+ * of a timer cannot be changed once started - a timer first needs to
+ * be stopped before it can be started with another timer expiration.
+ *
+ * Examples of use: watchdog timeout on job execution duration,
+ * a job progress checker, power management profile timeouts.
+ *
+ * @{
+ */
+
+/**
+ * @brief Initializes a timer
+ *
+ * Initializes a timer. The timer is not started yet.
+ *
+ * Note For timers created on stack @ref osk_timer_on_stack_init() should be used.
+ *
+ * The timer may be reinitialized but only after having called osk_timer_term()
+ * on \a tim.
+ *
+ * It is a programming error to pass a NULL pointer for \a tim. This will raise
+ * an assertion in debug builds.
+ *
+ * @param[out] tim an osk timer object to initialize
+ * @return OSK_ERR_NONE on success. Any other value indicates failure.
+ */
+OSK_STATIC_INLINE osk_error osk_timer_init(osk_timer * const tim) CHECK_RESULT;
+
+/**
+ * @brief Initializes a timer on stack
+ *
+ * Initializes a timer created on stack. The timer is not started yet.
+ *
+ * The timer may be reinitialized but only after having called osk_timer_term()
+ * on \a tim.
+ *
+ * It is a programming error to pass a NULL pointer for \a tim. This will raise
+ * an assertion in debug builds.
+ *
+ * @param[out] tim an osk timer object to initialize
+ * @return OSK_ERR_NONE on success. Any other value indicates failure.
+ */
+OSK_STATIC_INLINE osk_error osk_timer_on_stack_init(osk_timer * const tim) CHECK_RESULT;
+
+/**
+ * @brief Starts a timer
+ *
+ * Starts a timer. When the timer expires in \a delay milliseconds, the
+ * registered callback function will be called with the user supplied
+ * argument.
+ *
+ * The callback needs to be registered with osk_timer_callback_set()
+ * at least once during the lifetime of timer \a tim and before starting
+ * the timer.
+ *
+ * You cannot start a timer that has been started already. A timer needs
+ * to be stopped before starting it again.
+ *
+ * A timer cannot expire immediately. The minimum \a delay value is 1.
+ *
+ * A timer may fail to start and is important to check the result of this
+ * function to prevent waiting for a callback that will never get called.
+ *
+ * It is a programming error to pass a NULL pointer for \a tim or to specify
+ * 0 for \a delay. This will raise an assertion in debug builds.
+ *
+ * @param[in] tim an initialized osk timer object
+ * @param[in] delay timer expiration in milliseconds, at least 1.
+ * @return OSK_ERR_NONE on success. Any other value indicates failure.
+ */
+OSK_STATIC_INLINE osk_error osk_timer_start(osk_timer *tim, u32 delay) CHECK_RESULT;
+
+/**
+ * @brief Starts a timer using a high-resolution parameter
+ *
+ * This is identical to osk_timer_start(), except that the argument is
+ * expressed in nanoseconds.
+ *
+ * @note whilst the parameter is high-resolution, the actual resolution of the
+ * timer may be much more coarse than nanoseconds. In this case, \a delay_ns
+ * will be rounded up to the timer resolution.
+ *
+ * It is a programming error to pass a NULL pointer for \a tim or to specify
+ * 0 for \a delay_ns. This will raise an assertion in debug builds.
+ *
+ * @param[in] tim an initialized osk timer object
+ * @param[in] delay_ns timer expiration in nanoseconds, at least 1.
+ * @return OSK_ERR_NONE on success. Any other value indicates failure.
+ */
+OSK_STATIC_INLINE osk_error osk_timer_start_ns(osk_timer *tim, u64 delay_ns) CHECK_RESULT;
+
+/**
+ * @brief Modifies a timer's timeout
+ *
+ * See \a osk_timer_start for details.
+ *
+ * The only difference of this function from \a osk_timer_start is:
+ * If the timer was already set to expire the timer is modified to expire in \a new_delay milliseconds.
+ *
+ * It is a programming error to pass a NULL pointer for \a tim or to specify
+ * 0 for \a new_delay. This will raise an assertion in debug builds.
+ *
+ * @param[in] tim an initialized osk timer object
+ * @param[in] new_delay timer expiration in milliseconds, at least 1.
+ * @return OSK_ERR_NONE on success. Any other value indicates failure.
+ */
+OSK_STATIC_INLINE osk_error osk_timer_modify(osk_timer *tim, u32 new_delay) CHECK_RESULT;
+
+/**
+ * @brief Modifies a timer's timeout using a high-resolution parameter
+ *
+ * This is identical to osk_timer_modify(), except that the argument is
+ * expressed in nanoseconds.
+ *
+ * @note whilst the parameter is high-resolution, the actual resolution of the
+ * timer may be much more coarse than nanoseconds. In this case, \a new_delay_ns
+ * will be rounded up to the timer resolution.
+ *
+ * It is a programming error to pass a NULL pointer for \a tim or to specify
+ * 0 for \a new_delay_ns. This will raise an assertion in debug builds.
+ *
+ * @param[in] tim an initialized osk timer object
+ * @param[in] new_delay_ns timer expiration in nanoseconds, at least 1.
+ * @return OSK_ERR_NONE on success. Any other value indicates failure.
+ */
+OSK_STATIC_INLINE osk_error osk_timer_modify_ns(osk_timer *tim, u64 new_delay_ns) CHECK_RESULT;
+
+/**
+ * @brief Stops a timer
+ *
+ * Stops a timer. If the timer already expired this will have no
+ * effect. If the callback for the timer is currently executing,
+ * this function will block on its completion.
+ *
+ * A non-expired timer will have to be stopped before it can be
+ * started again with osk_timer_start().
+ *
+ * It is a programming error to pass a NULL pointer for \a tim. This will raise
+ * an assertion in debug builds.
+ *
+ * @param[in] tim an initialized osk timer object
+ */
+OSK_STATIC_INLINE void osk_timer_stop(osk_timer *tim);
+
+/**
+ * @brief Registers a callback function with a timer
+ *
+ * Registers a callback function to be called when the timer expires.
+ * The callback function is called with the provided \a data argument.
+ * The timer should be stopped when registering the callback.
+ *
+ * The code executing within the timer call back is limited as it
+ * is assumed it executes in an IRQ context:
+ * - Access to user space is not allowed - there is no process context
+ * - It is not allowed to call any function that may block
+ * - Only spinlocks or atomics may be used to access shared data structures
+ *
+ * If a timer requires more work to be done than can be achieved in an IRQ
+ * context, then it should defer the work to an OSK workqueue.
+ *
+ * It is a programming error to pass a NULL pointer for \a tim, \a callback.
+ * This will raise an assertion in debug builds. NULL is allowed for \a data.
+ *
+ * @param[in] tim an initialized osk timer object
+ * @param[in] callback timer callback function
+ * @param[in] data argument to pass to timer callback
+ */
+OSK_STATIC_INLINE void osk_timer_callback_set(osk_timer *tim, osk_timer_callback callback, void *data);
+
+/**
+ * @brief Terminates a timer
+ *
+ * Frees any resources allocated for a timer. A timer needs to be
+ * stopped with osk_timer_stop() before a timer can be terminated.
+ *
+ * Note For timers created on stack @ref osk_timer_on_stack_term() should be used.
+ *
+ * A timer may be reinitialized after it has been terminated.
+ *
+ * It is a programming error to pass a NULL pointer for \a tim. This will raise
+ * an assertion in debug builds.
+ *
+ * @param[in] tim an initialized osk timer object
+ */
+OSK_STATIC_INLINE void osk_timer_term(osk_timer *tim);
+
+/**
+ * @brief Terminates a timer on stack
+ *
+ * Frees any resources allocated for a timer created on stack. A timer needs to be
+ * stopped with osk_timer_stop() before a timer can be terminated.
+ *
+ * A timer may be reinitialized after it has been terminated.
+ *
+ * It is a programming error to pass a NULL pointer for \a tim. This will raise
+ * an assertion in debug builds.
+ *
+ * @param[in] tim an initialized osk timer object
+ */
+OSK_STATIC_INLINE void osk_timer_on_stack_term(osk_timer *tim);
+
+/* @} */ /* end group osktimers */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+/* pull in the arch header with the implementation */
+#include <osk/mali_osk_arch_timers.h>
+
+#ifdef __cplusplus
+}
+#endif
+
+
+#endif /* _OSK_TIMERS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _OSK_TYPES_H_
+#define _OSK_TYPES_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+typedef enum osk_error
+{
+ OSK_ERR_NONE, /**< Success */
+ OSK_ERR_FAIL, /**< Unclassified failure */
+ OSK_ERR_MAP, /**< Memory mapping operation failed */
+ OSK_ERR_ALLOC, /**< Memory allocation failed */
+ OSK_ERR_ACCESS /**< Permissions to access an object failed */
+} osk_error;
+
+#define OSK_STATIC_INLINE static __inline
+#define OSK_BITS_PER_LONG (8 * sizeof(unsigned long))
+#define OSK_ULONG_MAX (~0UL)
+
+/**
+ * OSK workqueue flags
+ *
+ * Flags specifying the kind of workqueue to create. Flags can be combined.
+ */
+
+/**
+ * By default a work queue guarantees non-reentrace on the same CPU.
+ * When the OSK_WORKQ_NON_REENTRANT flag is set, this guarantee is
+ * extended to all CPUs.
+ */
+#define OSK_WORKQ_NON_REENTRANT (1 << 0)
+/**
+ * Work units submitted to a high priority queue start execution as soon
+ * as resources are available.
+ */
+#define OSK_WORKQ_HIGH_PRIORITY (1 << 1)
+/**
+ * Ensures there is always a thread available to run tasks on this queue. This
+ * flag should be set if the work queue is involved in reclaiming memory when
+ * its work units run.
+ */
+#define OSK_WORKQ_RESCUER (1 << 2)
+
+/**
+ * Prototype for a function called when a OSK timer expires. See osk_timer_callback_set()
+ * that registers the callback function with a OSK timer.
+ */
+typedef void (*osk_timer_callback)(void *);
+
+typedef enum osk_power_state
+{
+ OSK_POWER_STATE_OFF, /**< Device is off */
+ OSK_POWER_STATE_IDLE, /**< Device is idle */
+ OSK_POWER_STATE_ACTIVE /**< Device is active */
+} osk_power_state;
+
+typedef enum osk_power_request_result
+{
+ OSK_POWER_REQUEST_FINISHED, /**< The driver successfully completed the power state change for the device */
+ OSK_POWER_REQUEST_FAILED, /**< The driver for the device encountered an error changing the power state */
+ OSK_POWER_REQUEST_REFUSED /**< The OS didn't allow the power state change for the device */
+} osk_power_request_result;
+
+
+#include <osk/mali_osk_arch_types.h>
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _OSK_TYPES_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_waitq.h
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_WAITQ_H_
+#define _OSK_WAITQ_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @addtogroup oskwaitq Wait queue
+ *
+ * A waitqueue is used to wait for a specific condition to become true.
+ * The waitqueue has a flag that needs to be set when the condition
+ * becomes true and cleared when the condition becomes false.
+ *
+ * Threads wait for the specific condition to become true by calling
+ * osk_waitq_wait(). If the condition is already true osk_waitq_wait()
+ * will return immediately.
+ *
+ * When a thread causes the specific condition to become true, it needs
+ * to set the waitqueue flag with osk_waitq_set(), which will wakeup
+ * all threads waiting on the waitqueue.
+ *
+ * When a thread causes the specific condition to become false, it needs
+ * to clear the waitqueue flag with osk_waitq_clear().
+ *
+ * @{
+ */
+
+/**
+ * @brief Initialize a wait queue
+ *
+ * Initializes a waitqueue. The waitqueue flag is cleared assuming the
+ * specific condition associated with the waitqueue is false.
+ *
+ * @param[out] wq wait queue to initialize
+ * @return OSK_ERROR_NONE on success. Any other value indicates failure.
+ */
+OSK_STATIC_INLINE osk_error osk_waitq_init(osk_waitq * const wq) CHECK_RESULT;
+
+/**
+ * @brief Wait until waitqueue flag is set
+ *
+ * Blocks until a thread signals the waitqueue that the condition has
+ * become true. Use osk_waitq_set() to set the waitqueue flag to signal
+ * the condition has become true. If the condition is already true,
+ * this function will return immediately.
+ *
+ * @param[in] wq initialized waitqueue
+ */
+OSK_STATIC_INLINE void osk_waitq_wait(osk_waitq *wq);
+
+/**
+ * @brief Set the waitqueue flag
+ *
+ * Signals the waitqueue that the condition associated with the waitqueue
+ * has become true. All threads on the waitqueue will be woken up. The
+ * waitqueue flag is set.
+ *
+ * @param[in] wq initialized waitqueue
+ */
+OSK_STATIC_INLINE void osk_waitq_set(osk_waitq *wq);
+
+/**
+ * @brief Clear the waitqueue flag
+ *
+ * Signals the waitqueue that the condition associated with the waitqueue
+ * has become false. The waitqueue flag is reset (cleared).
+ *
+ * @param[in] wq initialized waitqueue
+ */
+OSK_STATIC_INLINE void osk_waitq_clear(osk_waitq *wq);
+
+/**
+ * @brief Terminate a wait queue
+ *
+ * Frees any resources allocated for a waitqueue.
+ *
+ * No threads are allowed to be waiting on the waitqueue when terminating
+ * the waitqueue. If there are waiting threads, they should be woken up
+ * first by setting the waitqueue flag with osk_waitq_set() after which
+ * they must cease using the waitqueue.
+ *
+ * @param[in] wq initialized waitqueue
+ */
+OSK_STATIC_INLINE void osk_waitq_term(osk_waitq *wq);
+
+/* @} */ /* end group oskwaitq */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+/* pull in the arch header with the implementation */
+#include <osk/mali_osk_arch_waitq.h>
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _OSK_WAITQ_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _OSK_WORKQ_H
+#define _OSK_WORKQ_H
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+/* pull in the arch header with the implementation */
+#include <osk/mali_osk_arch_workq.h>
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @addtogroup base_osk_api
+ * @{
+ */
+
+/**
+ * @addtogroup oskworkqueue Work queue
+ *
+ * A workqueue is a queue of functions that will be invoked by one or more worker threads
+ * at some future time. Functions are invoked in FIFO order by each worker thread. However,
+ * overall execution of work is <b>not guaranteed to occur in FIFO order</b>, because two or
+ * more worker threads may be processing items concurrently from the same work queue.
+ *
+ * Each function that is submitted to the workqueue needs to be represented by a work unit
+ * (osk_workq_work). When a function is invoked, a pointer to the work unit is passed to the
+ * invoked function. A work unit needs to be embedded within the object that the invoked
+ * function needs to operate on, so that the invoked function can determine a pointer to the
+ * object it needs to operate on.
+ *
+ * @{
+ */
+
+/**
+ * @brief Initialize a work queue
+ *
+ * Initializes an empty work queue. One or more threads within the system will
+ * be servicing the work units submitted to the work queue.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL) for the
+ * wq parameter. Passing NULL will assert in debug builds.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL) for the
+ * name parameter. Passing NULL will assert in debug builds.
+ *
+ * It is a programming error to pass a value for flags other than a combination
+ * of the OSK_WORK_ constants or 0. Doing so will assert in debug builds.
+ *
+ * @param[out] wq workqueue to initialize
+ * @param[in] name The name for the queue (may be visible in the process list)
+ * @param[in] flags flags specifying behavior of work queue, see OSK_WORKQ_ constants.
+ * @return OSK_ERR_NONE on success. Any other value indicates failure.
+ */
+OSK_STATIC_INLINE osk_error osk_workq_init(osk_workq * const wq, const char *name, u32 flags) CHECK_RESULT;
+
+/**
+ * @brief Terminate a work queue
+ *
+ * Stops accepting new work and waits until the work queue is empty and
+ * all work has been completed, then frees any resources allocated for the workqueue.
+ *
+ * @param[in] wq intialized workqueue
+ */
+OSK_STATIC_INLINE void osk_workq_term(osk_workq *wq);
+
+/**
+ * @brief (Re)initialize a work object
+ *
+ * Sets up a work object to call the given function pointer.
+ * See \a osk_workq_work_init_on_stack if the work object
+ * is a stack object
+ * The function \a fn needs to match the prototype: void fn(osk_workq_work *).
+ *
+ * It is a programming error to pass an invalid pointer (including NULL) for
+ * any parameter. Passing NULL will assert in debug builds.
+ *
+ * @param[out] wk work unit to be initialized
+ * @param[in] fn function to be invoked at some future time
+ */
+OSK_STATIC_INLINE void osk_workq_work_init(osk_workq_work * const wk, osk_workq_fn fn);
+
+/**
+ * @brief (Re)initialize a work object allocated on the stack
+ *
+ * Sets up a work object to call the given function pointer.
+ * Special version needed for work objects on the stack.
+ * The function \a fn needs to match the prototype: void fn(osk_workq_work *).
+ *
+ * It is a programming error to pass an invalid pointer (including NULL) for
+ * any parameter. Passing NULL will assert in debug builds.
+ *
+ * @param[out] wk work unit to be initialized
+ * @param[in] fn function to be invoked at some future time
+ */
+OSK_STATIC_INLINE void osk_workq_work_init_on_stack(osk_workq_work * const wk, osk_workq_fn fn);
+
+
+/**
+ * @brief Submit work to a work queue
+ *
+ * Adds work (a work unit) to a work queue.
+ *
+ * The work unit (osk_workq_work) represents a function \a fn to be invoked at some
+ * future time. The invoked function \a fn is set via \a osk_workq_work_init or
+ * \a osk_workq_work_init_on_stack if the work object resides on the stack.
+ *
+ * The work unit should be embedded within the object that the invoked function needs
+ * to operate on, so that the invoked function can determine a pointer to the object
+ * it needs to operate on.
+ *
+ * osk_workq_submit() must be callable from IRQ context (it may not block nor access user space)
+ *
+ * The work unit memory \a wk needs to remain allocated until the function \a fn has been invoked.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL) for
+ * any parameter. Passing NULL will assert in debug builds.
+ *
+ * @param[in] wq intialized workqueue
+ * @param[out] wk initialized work object to submit
+ */
+OSK_STATIC_INLINE void osk_workq_submit(osk_workq *wq, osk_workq_work * const wk);
+
+/**
+ * @brief Flush a work queue
+ *
+ * All work units submitted to \a wq before this call will be complete by the
+ * time this function returns. The work units are guaranteed to be completed
+ * across the pool of worker threads.
+ *
+ * However, if a thread submits new work units to \a wq during the flush, then
+ * this function will not prevent those work units from running, nor will it
+ * guarantee to wait until after those work units are complete.
+ *
+ * Providing that no other thread attempts to submit work units to \a wq during
+ * or after this call, then it is guaranteed that no worker thread is executing
+ * any work from \a wq.
+ *
+ * @note The caller must ensure that they hold no locks that are also obtained
+ * by any work units on \a wq. Otherwise, a deadlock \b will occur.
+ *
+ * @note In addition, you must never call osk_workq_flush() from within any
+ * work unit, since this would cause a deadlock. Whilst it would normally be
+ * possible for a work unit to flush a different work queue, this may still
+ * cause a deadlock when the underlying implementation is using a single
+ * work queue for all work queues in the system.
+ *
+ * It is a programming error to pass an invalid pointer (including NULL) for
+ * any parameter. Passing NULL will assert in debug builds.
+ *
+ * @param[in] wq intialized workqueue to flush
+ */
+OSK_STATIC_INLINE void osk_workq_flush(osk_workq *wq);
+
+/** @} */ /* end group oskworkqueue */
+
+/** @} */ /* end group base_osk_api */
+
+/** @} */ /* end group base_api */
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _OSK_WORKQ_H */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#ifndef _OSK_H_
+#define _OSK_H_
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @defgroup base_osk_api Kernel-side OSK APIs
+ */
+
+/** @} */ /* end group base_api */
+
+#include "include/mali_osk_types.h"
+#include "include/mali_osk_debug.h"
+#if (1== MALI_BASE_TRACK_MEMLEAK)
+#include "include/mali_osk_failure.h"
+#endif
+#include "include/mali_osk_math.h"
+#include "include/mali_osk_lists.h"
+#include "include/mali_osk_lock_order.h"
+#include "include/mali_osk_locks.h"
+#include "include/mali_osk_atomics.h"
+#include "include/mali_osk_timers.h"
+#include "include/mali_osk_time.h"
+#include "include/mali_osk_bitops.h"
+#include "include/mali_osk_workq.h"
+#include "include/mali_osk_mem.h"
+#include "include/mali_osk_low_level_mem.h"
+#include "include/mali_osk_waitq.h"
+#include "include/mali_osk_power.h"
+#include "include/mali_osk_credentials.h"
+
+#endif /* _OSK_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_common.h
+ * This header defines macros shared by the Common Layer public interfaces for
+ * all utilities, to ensure they are available even if a client does not include
+ * the convenience header mali_osk.h.
+ */
+
+#ifndef _OSK_COMMON_H_
+#define _OSK_COMMON_H_
+
+#include <osk/include/mali_osk_debug.h>
+
+/**
+ * @private
+ */
+static INLINE mali_bool oskp_ptr_is_null(const void* ptr)
+{
+ CSTD_UNUSED(ptr);
+ OSK_ASSERT(ptr != NULL);
+ return MALI_FALSE;
+}
+
+/**
+ * @brief Check if a pointer is NULL, if so an assert is triggered, otherwise the pointer itself is returned.
+ *
+ * @param [in] ptr Pointer to test
+ *
+ * @return @c ptr if it's not NULL.
+ */
+#define OSK_CHECK_PTR(ptr)\
+ (oskp_ptr_is_null(ptr) ? NULL : ptr)
+
+#endif /* _OSK_COMMON_H_ */
--- /dev/null
+obj-y += linux/
+obj-y += common/
--- /dev/null
+ccflags-$(CONFIG_VITHAR) += -DMALI_DEBUG=1 -DMALI_HW_TYPE=2 \
+-DMALI_USE_UMP=0 -DMALI_HW_VERSION=r0p0 -DMALI_BASE_TRACK_MEMLEAK=0 \
+-DMALI_ANDROID=1 -DMALI_ERROR_INJECT_ON=0 -DMALI_NO_MALI=0 -DMALI_BACKEND_KERNEL=1 \
+-DMALI_FAKE_PLATFORM_DEVICE=1 -DMALI_MOCK_TEST=0 -DMALI_KERNEL_TEST_API=0 \
+-DMALI_INFINITE_CACHE=0 -DMALI_LICENSE_IS_GPL=1 -DMALI_PLATFORM_CONFIG=exynos5 \
+-DMALI_UNIT_TEST=0 -DMALI_GATOR_SUPPORT=0 -DUMP_LICENSE_IS_GPL=1 \
+-DUMP_SVN_REV_STRING="\"dummy\"" -DMALI_RELEASE_NAME="\"dummy\""
+
+ROOTDIR = $(src)/../../..
+
+ccflags-$(CONFIG_VITHAR) += -I$(ROOTDIR)/kbase
+ccflags-$(CONFIG_VITHAR) += -I$(ROOTDIR)
+
+ccflags-y += -I$(ROOTDIR) -I$(ROOTDIR)/include -I$(ROOTDIR)/osk/src/linux/include -I$(ROOTDIR)/uk/platform_dummy
+ccflags-y += -I$(ROOTDIR)/kbase/midg_gpus/r0p0
+
+
+obj-y += mali_osk_bitops_cmn.o mali_osk_debug_cmn.o
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+
+#include <osk/mali_osk.h>
+
+unsigned long osk_bitarray_find_first_zero_bit(const unsigned long *addr, unsigned long maxbit)
+{
+ unsigned long total;
+
+ OSK_ASSERT(NULL != addr);
+
+ for ( total = 0; total < maxbit; total += OSK_BITS_PER_LONG, ++addr )
+ {
+ if (OSK_ULONG_MAX != *addr)
+ {
+ int result;
+ result = oskp_find_first_zero_bit( *addr );
+ /* non-negative signifies the bit was found */
+ if ( result >= 0 )
+ {
+ total += (unsigned long)result;
+ break;
+ }
+ }
+ }
+
+ /* Now check if we reached maxbit or above */
+ if ( total >= maxbit )
+ {
+ total = maxbit;
+ }
+
+ return total; /* either the found bit nr, or maxbit if not found */
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file osk/src/common/mali_osk_compile_asserts.h
+ *
+ * Private definitions of compile time asserts.
+ **/
+
+/**
+ * Unreachable function needed to check values at compile-time, in both debug
+ * and release builds
+ */
+void oskp_cmn_compile_time_assertions(void);
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+
+#include <osk/mali_osk.h>
+#include "mali_osk_compile_asserts.h"
+
+/**
+ * @brief Contains the module names (modules in the same order as for the osk_module enumeration)
+ * @sa oskp_module_to_str
+ */
+static const char* CONST oskp_str_modules[] =
+{
+ "UNKNOWN", /**< Unknown module */
+ "OSK", /**< OSK */
+ "UKK", /**< UKK */
+ "BASE_MMU", /**< Base MMU */
+ "BASE_JD", /**< Base Job Dispatch */
+ "BASE_JM", /**< Base Job Manager */
+ "BASE_CORE", /**< Base Core */
+ "BASE_MEM", /**< Base Memory */
+ "BASE_EVENT", /**< Base Event */
+ "BASE_CTX", /**< Base Context */
+ "BASE_PM", /**< Base Power Management */
+ "UMP", /**< UMP */
+};
+
+#define MODULE_STRING_ARRAY_SIZE (sizeof(oskp_str_modules)/sizeof(oskp_str_modules[0]))
+
+INLINE void oskp_cmn_compile_time_assertions(void)
+{
+ /*
+ * If this assert triggers you have forgotten to update oskp_str_modules
+ * when you added a module to the osk_module enum
+ * */
+ CSTD_COMPILE_TIME_ASSERT(OSK_MODULES_ALL == MODULE_STRING_ARRAY_SIZE );
+}
+
+const char* oskp_module_to_str(const osk_module module)
+{
+ if( MODULE_STRING_ARRAY_SIZE <= module)
+ {
+ return "";
+ }
+ return oskp_str_modules[module];
+}
+
+void oskp_validate_format_string(const char *format, ...)
+{
+#if MALI_DEBUG
+ char c;
+ static const char *supported[] =
+ {
+ "d", "ld", "lld",
+ "x", "lx", "llx",
+ "X", "lX", "llX",
+ "u", "lu", "llu",
+ "p",
+ "c",
+ "s",
+ };
+ static const unsigned char sizes[] = { 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 1, 1 };
+
+ unsigned int i;
+
+ /* %[flags][width][.precision][length]specifier */
+
+ while ( (c = *format++) )
+ {
+ if (c == '%')
+ {
+ c = *format;
+
+ if (c == '\0')
+ {
+ /* Unsupported format */
+ OSK_PRINT_WARN(OSK_OSK, "OSK Format specification not complete (%% not followed by anything)\n");
+ return;
+ }
+ else if (c != '%')
+ {
+ /* Skip to the [length]specifier part assuming it starts with
+ * an alphabetic character and flags, width, precision do not
+ * contain alphabetic characters.
+ */
+ do
+ {
+ if ((c >= 'a' && c <= 'z') || c == 'X')
+ {
+ /* Match supported formats with current position in format string */
+ for (i = 0; i < NELEMS(supported); i++)
+ {
+ if (strncmp(format, supported[i], sizes[i]) == 0)
+ {
+ /* Supported format */
+ break;
+ }
+ }
+
+ if (i == NELEMS(supported))
+ {
+ /* Unsupported format */
+ OSK_PRINT_WARN(OSK_OSK, "OSK Format string specifier not supported (starting at '%s')\n", format);
+ return;
+ }
+
+ /* Start looking for next '%' */
+ break;
+ }
+ } while ( (c = *++format) );
+ }
+ }
+ }
+#else
+ CSTD_UNUSED(format);
+#endif
+}
--- /dev/null
+ccflags-$(CONFIG_VITHAR) += -DMALI_DEBUG=1 -DMALI_HW_TYPE=2 \
+ -DMALI_USE_UMP=0 -DMALI_HW_VERSION=r0p0 -DMALI_BASE_TRACK_MEMLEAK=0 \
+ -DMALI_ANDROID=1 -DMALI_ERROR_INJECT_ON=0 -DMALI_NO_MALI=0 -DMALI_BACKEND_KERNEL=1 \
+ -DMALI_FAKE_PLATFORM_DEVICE=1 -DMALI_MOCK_TEST=0 -DMALI_KERNEL_TEST_API=0 \
+ -DMALI_INFINITE_CACHE=0 -DMALI_LICENSE_IS_GPL=1 -DMALI_PLATFORM_CONFIG=exynos5 \
+ -DMALI_UNIT_TEST=0 -DMALI_GATOR_SUPPORT=0 -DUMP_LICENSE_IS_GPL=1 \
+ -DUMP_SVN_REV_STRING="\"dummy\"" -DMALI_RELEASE_NAME="\"dummy\""
+
+ROOTDIR = $(src)/../../..
+
+ccflags-$(CONFIG_VITHAR) += -I$(ROOTDIR)/kbase
+ccflags-$(CONFIG_VITHAR) += -I$(ROOTDIR)
+
+ccflags-y += -I$(ROOTDIR) -I$(ROOTDIR)/include -I$(ROOTDIR)/osk/src/linux/include -I$(ROOTDIR)/uk/platform_dummy
+ccflags-y += -I$(ROOTDIR)/kbase/midg_gpus/r0p0
+
+obj-y += mali_osk_timers.o mali_osk_debug.o
--- /dev/null
+# Copyright:
+# ----------------------------------------------------------------------------
+# This confidential and proprietary software may be used only as authorized
+# by a licensing agreement from ARM Limited.
+# (C) COPYRIGHT 2010 ARM Limited, ALL RIGHTS RESERVED
+# The entire notice above must be reproduced on all authorized copies and
+# copies may only be made to the extent permitted by a licensing agreement
+# from ARM Limited.
+# ----------------------------------------------------------------------------
+#
+
+EXTRA_CFLAGS += -I$(ROOT) -I$(ROOT)/osk/src/linux/include
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_ATOMICS_H_
+#define _OSK_ARCH_ATOMICS_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+OSK_STATIC_INLINE u32 osk_atomic_sub(osk_atomic * atom, u32 value)
+{
+ OSK_ASSERT(NULL != atom);
+ return atomic_sub_return(value, atom);
+}
+
+OSK_STATIC_INLINE u32 osk_atomic_add(osk_atomic * atom, u32 value)
+{
+ OSK_ASSERT(NULL != atom);
+ return atomic_add_return(value, atom);
+}
+
+OSK_STATIC_INLINE u32 osk_atomic_dec(osk_atomic * atom)
+{
+ OSK_ASSERT(NULL != atom);
+ return osk_atomic_sub(atom, 1);
+}
+
+OSK_STATIC_INLINE u32 osk_atomic_inc(osk_atomic * atom)
+{
+ OSK_ASSERT(NULL != atom);
+ return osk_atomic_add(atom, 1);
+}
+
+OSK_STATIC_INLINE void osk_atomic_set(osk_atomic * atom, u32 value)
+{
+ OSK_ASSERT(NULL != atom);
+ atomic_set(atom, value);
+}
+
+OSK_STATIC_INLINE u32 osk_atomic_get(osk_atomic * atom)
+{
+ OSK_ASSERT(NULL != atom);
+ return atomic_read(atom);
+}
+
+OSK_STATIC_INLINE u32 osk_atomic_compare_and_swap(osk_atomic * atom, u32 old_value, u32 new_value)
+{
+ OSK_ASSERT(NULL != atom);
+ return atomic_cmpxchg(atom, old_value, new_value);
+}
+
+#endif /* _OSK_ARCH_ATOMICS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_BITOPS_H_
+#define _OSK_ARCH_BITOPS_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#include <linux/bitops.h>
+
+OSK_STATIC_INLINE long osk_clz(unsigned long val)
+{
+ return OSK_BITS_PER_LONG - fls_long(val);
+}
+
+OSK_STATIC_INLINE long osk_clz_64(u64 val)
+{
+ return 64 - fls64(val);
+}
+
+OSK_STATIC_INLINE int osk_count_set_bits(unsigned long val)
+{
+ /* note: __builtin_popcountl() not available in kernel */
+ int count = 0;
+ while (val)
+ {
+ count++;
+ val &= (val-1);
+ }
+ return count;
+}
+
+#endif /* _OSK_ARCH_BITOPS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_CREDENTIALS_H_
+#define _OSK_ARCH_CREDENTIALS_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#include <linux/cred.h>
+
+OSK_STATIC_INLINE mali_bool osk_is_privileged(void)
+{
+ mali_bool is_privileged = MALI_FALSE;
+
+ /* Check if the caller is root */
+ if (current_euid() == 0)
+ {
+ is_privileged = MALI_TRUE;
+ }
+
+ return is_privileged;
+}
+
+OSK_STATIC_INLINE mali_bool osk_is_policy_realtime(void)
+{
+ int policy = current->policy;
+
+ if (policy == SCHED_FIFO || policy == SCHED_RR)
+ {
+ return MALI_TRUE;
+ }
+
+ return MALI_FALSE;
+}
+
+OSK_STATIC_INLINE void osk_get_process_priority(osk_process_priority *prio)
+{
+ /* Note that we return the current process priority.
+ * If called from a kernel thread the priority returned
+ * will be the kernel thread priority and not the user
+ * process that is currently submitting jobs to the scheduler.
+ */
+ OSK_ASSERT(prio);
+
+ if(osk_is_policy_realtime())
+ {
+ prio->is_realtime = MALI_TRUE;
+ /* NOTE: realtime range was in the range 0..99 (lowest to highest) so we invert
+ * the priority and scale to -20..0 to normalize the result with the NICE range
+ */
+ prio->priority = (((MAX_RT_PRIO-1) - current->rt_priority) / 5) - 20;
+ /* Realtime range returned:
+ * -20 - highest priority
+ * 0 - lowest priority
+ */
+ }
+ else
+ {
+ prio->is_realtime = MALI_FALSE;
+ prio->priority = (current->static_prio - MAX_RT_PRIO) - 20;
+ /* NICE range returned:
+ * -20 - highest priority
+ * +19 - lowest priority
+ */
+ }
+}
+
+#endif /* _OSK_ARCH_CREDENTIALS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_DEBUG_H_
+#define _OSK_ARCH_DEBUG_H_
+
+#include <malisw/mali_stdtypes.h>
+#include "mali_osk_arch_types.h"
+
+#if MALI_UNIT_TEST
+/* Kernel testing helpers */
+void osk_kernel_test_init(void);
+void osk_kernel_test_term(void);
+void osk_kernel_test_wait(void);
+void osk_kernel_test_signal(void);
+mali_bool osk_kernel_test_has_asserted(void);
+void oskp_kernel_test_exit(void);
+#endif
+
+/** Maximum number of bytes (incl. end of string character) supported in the generated debug output string */
+#define OSK_DEBUG_MESSAGE_SIZE 256
+
+/**
+ * All OSKP_ASSERT* and OSKP_PRINT_* macros will eventually call OSKP_PRINT to output messages
+ */
+void oskp_debug_print(const char *fmt, ...);
+#define OSKP_PRINT(...) oskp_debug_print(__VA_ARGS__)
+
+/**
+ * Insert a breakpoint to cause entry in an attached debugger. However, since there is
+ * no API available to trigger entry in a debugger, we dereference a NULL
+ * pointer which should cause an exception and enter a debugger.
+ */
+#define OSKP_BREAKPOINT() *(int *)0 = 0
+
+/**
+ * Quit the driver and halt.
+ */
+#define OSKP_QUIT() BUG()
+
+/**
+ * Print a backtrace
+ */
+#define OSKP_TRACE() WARN_ON(1)
+
+#define OSKP_CHANNEL_INFO ((u32)0x00000001) /**< @brief No output*/
+#define OSKP_CHANNEL_WARN ((u32)0x00000002) /**< @brief Standard output*/
+#define OSKP_CHANNEL_ERROR ((u32)0x00000004) /**< @brief Error output*/
+#define OSKP_CHANNEL_RAW ((u32)0x00000008) /**< @brief Raw output*/
+#define OSKP_CHANNEL_ALL ((u32)0xFFFFFFFF) /**< @brief All the channels at the same time*/
+
+/** @brief Disable the asserts tests if set to 1. Default is to enable the asserts. */
+#ifndef OSK_DISABLE_ASSERT
+#define OSK_DISABLE_ASSERTS 0
+#endif
+
+/** @brief If equals to 0, a trace containing the file, line, and function will be displayed before each message. */
+#define OSK_SKIP_TRACE 0
+
+/** @brief If different from 0, the trace will only contain the file and line. */
+#define OSK_SKIP_FUNCTION_NAME 0
+
+/** @brief Variable to set the permissions per module and per channel.
+ */
+#define OSK_MODULES_PERMISSIONS "ALL_ALL"
+
+/** @brief String terminating every message printed by the debug API */
+#define OSK_STOP_MSG "\n"
+
+/** @brief Enables support for runtime configuration if set to 1.
+ */
+#define OSK_USE_RUNTIME_CONFIG 0
+#define OSK_SIMULATE_FAILURES MALI_BASE_TRACK_MEMLEAK /**< @brief Enables simulation of failures (for testing) if non-zero */
+
+#define OSK_ACTION_IGNORE 0 /**< @brief The given message is ignored then the execution continues*/
+#define OSK_ACTION_PRINT_AND_CONTINUE 1 /**< @brief The given message is printed then the execution continues*/
+#define OSK_ACTION_PRINT_AND_BREAK 2 /**< @brief The given message is printed then a break point is triggered*/
+#define OSK_ACTION_PRINT_AND_QUIT 3 /**< @brief The given message is printed then the execution is stopped*/
+#define OSK_ACTION_PRINT_AND_TRACE 4 /**< @brief The given message and a backtrace is printed then the execution continues*/
+
+/**
+ * @def OSK_ON_INFO
+ * @brief Defines the API behavior when @ref OSK_PRINT_INFO() is called
+ * @note Must be set to one of the following values: @see OSK_ACTION_PRINT_AND_CONTINUE,
+ * @note @ref OSK_ACTION_PRINT_AND_BREAK, @see OSK_ACTION_PRINT_AND_QUIT, @see OSK_ACTION_IGNORE
+ *
+ * @def OSK_ON_WARN
+ * @brief Defines the API behavior when @see OSK_PRINT_WARN() is called
+ * @note Must be set to one of the following values: @see OSK_ACTION_PRINT_AND_CONTINUE,
+ * @note @see OSK_ACTION_PRINT_AND_BREAK, @see OSK_ACTION_PRINT_AND_QUIT, @see OSK_ACTION_IGNORE
+ *
+ * @def OSK_ON_ERROR
+ * @brief Defines the API behavior when @see OSK_PRINT_ERROR() is called
+ * @note Must be set to one of the following values: @see OSK_ACTION_PRINT_AND_CONTINUE,
+ * @note @see OSK_ACTION_PRINT_AND_BREAK, @see OSK_ACTION_PRINT_AND_QUIT, @see OSK_ACTION_IGNORE
+ *
+ * @def OSK_ON_ASSERT
+ * @brief Defines the API behavior when @see OSKP_PRINT_ASSERT() is called
+ * @note Must be set to one of the following values: @see OSK_ACTION_PRINT_AND_CONTINUE,
+ * @note @see OSK_ACTION_PRINT_AND_BREAK, @see OSK_ACTION_PRINT_AND_QUIT, @see OSK_ACTION_IGNORE
+ *
+ * @def OSK_ON_RAW
+ * @brief Defines the API behavior when @see OSKP_PRINT_RAW() is called
+ * @note Must be set to one of the following values: @see OSK_ACTION_PRINT_AND_CONTINUE,
+ * @note @see OSK_ACTION_PRINT_AND_BREAK, @see OSK_ACTION_PRINT_AND_QUIT, @see OSK_ACTION_IGNORE
+ *
+ *
+ */
+#if MALI_DEBUG
+ #define OSK_ON_INFO OSK_ACTION_IGNORE
+ #define OSK_ON_WARN OSK_ACTION_PRINT_AND_CONTINUE
+ #define OSK_ON_ASSERT OSK_ACTION_PRINT_AND_QUIT
+ #define OSK_ON_ERROR OSK_ACTION_PRINT_AND_CONTINUE
+ #define OSK_ON_RAW OSK_ACTION_PRINT_AND_CONTINUE
+#else
+ #define OSK_ON_INFO OSK_ACTION_IGNORE
+ #define OSK_ON_WARN OSK_ACTION_IGNORE
+ #define OSK_ON_ASSERT OSK_ACTION_IGNORE
+ #define OSK_ON_ERROR OSK_ACTION_PRINT_AND_CONTINUE
+ #define OSK_ON_RAW OSK_ACTION_PRINT_AND_CONTINUE
+#endif
+
+#if MALI_UNIT_TEST
+#define OSKP_KERNEL_TEST_ASSERT() oskp_kernel_test_exit()
+#else
+#define OSKP_KERNEL_TEST_ASSERT() CSTD_NOP()
+#endif
+
+/**
+ * OSK_ASSERT macros do nothing if the flag @see OSK_DISABLE_ASSERTS is set to 1
+ */
+#if OSK_DISABLE_ASSERTS
+ #define OSKP_ASSERT(expr) CSTD_NOP()
+ #define OSKP_INTERNAL_ASSERT(expr) CSTD_NOP()
+ #define OSKP_ASSERT_MSG(expr, ...) CSTD_NOP()
+#else /* OSK_DISABLE_ASSERTS */
+
+/**
+ * @def OSKP_ASSERT_MSG(expr, ...)
+ * @brief Calls @see OSKP_PRINT_ASSERT and prints the given message if @a expr is false
+ *
+ * @note This macro does nothing if the flag @see OSK_DISABLE_ASSERTS is set to 1
+ *
+ * @param expr Boolean expression
+ * @param ... Message to display when @a expr is false, as a format string followed by format arguments.
+ */
+#define OSKP_ASSERT_MSG(expr, ...)\
+ do\
+ {\
+ if(MALI_FALSE == (expr))\
+ {\
+ OSKP_PRINT_ASSERT(__VA_ARGS__);\
+ }\
+ }while(MALI_FALSE)
+
+/**
+ * @def OSKP_ASSERT(expr)
+ * @brief Calls @see OSKP_PRINT_ASSERT and prints the expression @a expr if @a expr is false
+ *
+ * @note This macro does nothing if the flag @see OSK_DISABLE_ASSERTS is set to 1
+ *
+ * @param expr Boolean expression
+ */
+#define OSKP_ASSERT(expr)\
+ OSKP_ASSERT_MSG(expr, #expr)
+
+/**
+ * @def OSKP_INTERNAL_ASSERT(expr)
+ * @brief Calls @see OSKP_BREAKPOINT if @a expr is false
+ * This assert function is for internal use of OSK functions which themselves are used to implement
+ * the OSK_ASSERT functionality. These functions should use OSK_INTERNAL_ASSERT which does not use
+ * any OSK functions to prevent ending up in a recursive loop.
+ *
+ * @note This macro does nothing if the flag @see OSK_DISABLE_ASSERTS is set to 1
+ *
+ * @param expr Boolean expression
+ */
+#define OSKP_INTERNAL_ASSERT(expr)\
+ do\
+ {\
+ if(MALI_FALSE == (expr))\
+ {\
+ OSKP_BREAKPOINT();\
+ }\
+ }while(MALI_FALSE)
+
+/**
+ * @def OSKP_PRINT_ASSERT(...)
+ * @brief Prints "MALI<ASSERT>" followed by trace, function name and the given message.
+ *
+ * The behavior of this function is defined by the macro @see OSK_ON_ASSERT.
+ *
+ * @note This macro does nothing if the flag @see OSK_DISABLE_ASSERTS is set to 1
+ *
+ * Example: OSKP_PRINT_ASSERT(" %d blocks could not be allocated", mem_alocated) will print:\n
+ * "MALI<ASSERT> In file <path> line: <line number> function:<function name> 10 blocks could not be allocated"
+ *
+ * @note Depending on the values of @see OSK_SKIP_FUNCTION_NAME and @see OSK_SKIP_TRACE the trace will be displayed
+ * before the message.
+ *
+ * @param ... Message to print, passed as a format string followed by format arguments.
+ */
+#define OSKP_PRINT_ASSERT(...)\
+ do\
+ {\
+ OSKP_ASSERT_OUT(OSKP_PRINT_TRACE, OSKP_PRINT_FUNCTION, __VA_ARGS__);\
+ oskp_debug_assert_call_hook();\
+ OSKP_KERNEL_TEST_ASSERT();\
+ OSKP_ASSERT_ACTION();\
+ }while(MALI_FALSE)
+
+#endif
+
+/**
+ * @def OSKP_DEBUG_CODE( X )
+ * @brief Executes the code inside the macro only in debug mode
+ *
+ * @param X Code to compile only in debug mode.
+ */
+#if MALI_DEBUG
+ #define OSKP_DEBUG_CODE( X ) X
+#else
+ #define OSKP_DEBUG_CODE( X ) CSTD_NOP()
+#endif
+
+/**
+ * @def OSKP_ASSERT_ACTION
+ * @brief (Private) Action associated to the @see OSKP_PRINT_ASSERT event.
+ */
+/* Configure the post display action */
+#if OSK_ON_ASSERT == OSK_ACTION_PRINT_AND_BREAK
+ #define OSKP_ASSERT_ACTION OSKP_BREAKPOINT
+#elif OSK_ON_ASSERT == OSK_ACTION_PRINT_AND_QUIT
+ #define OSKP_ASSERT_ACTION OSKP_QUIT
+#elif OSK_ON_ASSERT == OSK_ACTION_PRINT_AND_TRACE
+ #define OSKP_ASSERT_ACTION OSKP_TRACE
+#elif OSK_ON_ASSERT == OSK_ACTION_PRINT_AND_CONTINUE || OSK_ON_ASSERT == OSK_ACTION_IGNORE
+ #define OSKP_ASSERT_ACTION() CSTD_NOP()
+#else
+ #error invalid value for OSK_ON_ASSERT
+#endif
+
+/**
+ * @def OSKP_RAW_ACTION
+ * @brief (Private) Action associated to the @see OSK_PRINT_RAW event.
+ */
+/* Configure the post display action */
+#if OSK_ON_RAW == OSK_ACTION_PRINT_AND_BREAK
+ #define OSKP_RAW_ACTION OSKP_BREAKPOINT
+#elif OSK_ON_RAW == OSK_ACTION_PRINT_AND_QUIT
+ #define OSKP_RAW_ACTION OSKP_QUIT
+#elif OSK_ON_RAW == OSK_ACTION_PRINT_AND_TRACE
+ #define OSKP_RAW_ACTION OSKP_TRACE
+#elif OSK_ON_RAW == OSK_ACTION_PRINT_AND_CONTINUE || OSK_ON_RAW == OSK_ACTION_IGNORE
+ #define OSKP_RAW_ACTION() CSTD_NOP()
+#else
+ #error invalid value for OSK_ON_RAW
+#endif
+
+/**
+ * @def OSKP_INFO_ACTION
+ * @brief (Private) Action associated to the @see OSK_PRINT_INFO event.
+ */
+/* Configure the post display action */
+#if OSK_ON_INFO == OSK_ACTION_PRINT_AND_BREAK
+ #define OSKP_INFO_ACTION OSKP_BREAKPOINT
+#elif OSK_ON_INFO == OSK_ACTION_PRINT_AND_QUIT
+ #define OSKP_INFO_ACTION OSKP_QUIT
+#elif OSK_ON_INFO == OSK_ACTION_PRINT_AND_TRACE
+ #define OSKP_INFO_ACTION OSKP_TRACE
+#elif OSK_ON_INFO == OSK_ACTION_PRINT_AND_CONTINUE || OSK_ON_INFO == OSK_ACTION_IGNORE
+ #define OSKP_INFO_ACTION() CSTD_NOP()
+#else
+ #error invalid value for OSK_ON_INFO
+#endif
+
+/**
+ * @def OSKP_ERROR_ACTION
+ * @brief (Private) Action associated to the @see OSK_PRINT_ERROR event.
+ */
+/* Configure the post display action */
+#if OSK_ON_ERROR == OSK_ACTION_PRINT_AND_BREAK
+ #define OSKP_ERROR_ACTION OSKP_BREAKPOINT
+#elif OSK_ON_ERROR == OSK_ACTION_PRINT_AND_QUIT
+ #define OSKP_ERROR_ACTION OSKP_QUIT
+#elif OSK_ON_ERROR == OSK_ACTION_PRINT_AND_TRACE
+ #define OSKP_ERROR_ACTION OSKP_TRACE
+#elif OSK_ON_ERROR == OSK_ACTION_PRINT_AND_CONTINUE || OSK_ON_ERROR == OSK_ACTION_IGNORE
+ #define OSKP_ERROR_ACTION() CSTD_NOP()
+#else
+ #error invalid value for OSK_ON_ERROR
+#endif
+
+/**
+ * @def OSKP_WARN_ACTION
+ * @brief (Private) Action associated to the @see OSK_PRINT_WARN event.
+ */
+/* Configure the post display action */
+#if OSK_ON_WARN == OSK_ACTION_PRINT_AND_BREAK
+ #define OSKP_WARN_ACTION OSKP_BREAKPOINT
+#elif OSK_ON_WARN == OSK_ACTION_PRINT_AND_QUIT
+ #define OSKP_WARN_ACTION OSKP_QUIT
+#elif OSK_ON_WARN == OSK_ACTION_PRINT_AND_TRACE
+ #define OSKP_WARN_ACTION OSKP_TRACE
+#elif OSK_ON_WARN == OSK_ACTION_PRINT_AND_CONTINUE || OSK_ON_WARN == OSK_ACTION_IGNORE
+ #define OSKP_WARN_ACTION() CSTD_NOP()
+#else
+ #error invalid value for OSK_ON_WARN
+#endif
+
+/**
+ * @def OSKP_PRINT_RAW(module, ...)
+ * @brief Prints given message
+ *
+ * The behavior of this function is defined by macro @see OSK_ON_RAW
+ *
+ * Example:
+ * @code OSKP_PRINT_RAW(OSK_BASE_MEM, " %d blocks could not be allocated", mem_allocated); @endcode will print:
+ * \n
+ * "10 blocks could not be allocated"
+ *
+ * @param module Name of the module which prints the message.
+ * @param ... Format string followed by a varying number of parameters
+ */
+#define OSKP_PRINT_RAW(module, ...)\
+ do\
+ {\
+ if(MALI_FALSE != OSKP_PRINT_IS_ALLOWED( (module), OSK_CHANNEL_RAW))\
+ {\
+ OSKP_RAW_OUT(oskp_module_to_str( (module) ), \
+ OSKP_PRINT_TRACE, OSKP_PRINT_FUNCTION, __VA_ARGS__ );\
+ }\
+ OSKP_RAW_ACTION();\
+ }while(MALI_FALSE)
+
+/**
+ * @def OSKP_PRINT_INFO(module, ...)
+ * @brief Prints "MALI<INFO,module_name>: " followed by the given message.
+ *
+ * The behavior of this function is defined by the macro @see OSK_ON_INFO
+ *
+ * Example:
+ * @code OSKP_PRINT_INFO(OSK_BASE_MEM, " %d blocks could not be allocated", mem_allocated); @endcode will print:
+ * \n
+ * "MALI<INFO,BASE_MEM>: 10 blocks could not be allocated"\n
+ *
+ * @param module Name of the module which prints the message.
+ * @param ... Format string followed by a varying number of parameters
+ */
+#define OSKP_PRINT_INFO(module, ...)\
+ do\
+ {\
+ if(MALI_FALSE != OSKP_PRINT_IS_ALLOWED( (module), OSK_CHANNEL_INFO))\
+ {\
+ OSKP_INFO_OUT(oskp_module_to_str( (module) ), \
+ OSKP_PRINT_TRACE, OSKP_PRINT_FUNCTION, __VA_ARGS__ );\
+ }\
+ OSKP_INFO_ACTION();\
+ }while(MALI_FALSE)
+
+/**
+ * @def OSKP_PRINT_WARN(module, ...)
+ * @brief Prints "MALI<WARN,module_name>: " followed by the given message.
+ *
+ * The behavior of this function is defined by the macro @see OSK_ON_WARN
+ *
+ * Example:
+ * @code OSK_PRINT_WARN(OSK_BASE_MEM, " %d blocks could not be allocated", mem_allocated);@endcode will print: \n
+ * "MALI<WARN,BASE_MEM>: 10 blocks could not be allocated"\n
+ *
+ * @param module Name of the module which prints the message.
+ * @param ... Format string followed by a varying number of parameters
+ */
+#define OSKP_PRINT_WARN(module, ...)\
+ do\
+ {\
+ if(MALI_FALSE != OSKP_PRINT_IS_ALLOWED( (module), OSK_CHANNEL_WARN))\
+ {\
+ OSKP_WARN_OUT(oskp_module_to_str( (module) ), \
+ OSKP_PRINT_TRACE, OSKP_PRINT_FUNCTION, __VA_ARGS__ );\
+ }\
+ OSKP_WARN_ACTION();\
+ }while(MALI_FALSE)
+
+/**
+ * @def OSKP_PRINT_ERROR(module, ...)
+ * @brief Prints "MALI<ERROR,module_name>: " followed by the given message.
+ *
+ * The behavior of this function is defined by the macro @see OSK_ON_ERROR
+ *
+ * Example:
+ * @code OSKP_PRINT_ERROR(OSK_BASE_MEM, " %d blocks could not be allocated", mem_allocated); @endcode will print:
+ * \n
+ * "MALI<ERROR,BASE_MEM>: 10 blocks could not be allocated"\n
+ *
+ * @param module Name of the module which prints the message.
+ * @param ... Format string followed by a varying number of parameters
+ */
+#define OSKP_PRINT_ERROR(module, ...)\
+ do\
+ {\
+ if(MALI_FALSE != OSKP_PRINT_IS_ALLOWED( (module), OSK_CHANNEL_ERROR))\
+ {\
+ OSKP_ERROR_OUT(oskp_module_to_str( (module) ), \
+ OSKP_PRINT_TRACE, OSKP_PRINT_FUNCTION, __VA_ARGS__ );\
+ }\
+ OSKP_ERROR_ACTION();\
+ }while(MALI_FALSE)
+
+/**
+ * @def OSKP_PRINT_TRACE
+ * @brief Private macro containing the format of the trace to display before every message
+ * @sa OSK_SKIP_TRACE, OSK_SKIP_FUNCTION_NAME
+ */
+#if OSK_SKIP_TRACE == 0
+ #define OSKP_PRINT_TRACE \
+ "In file: " __FILE__ " line: " CSTD_STR2(__LINE__)
+ #if OSK_SKIP_FUNCTION_NAME == 0
+ #define OSKP_PRINT_FUNCTION CSTD_FUNC
+ #else
+ #define OSKP_PRINT_FUNCTION ""
+ #endif
+#else
+ #define OSKP_PRINT_TRACE ""
+#endif
+
+/**
+ * @def OSKP_PRINT_ALLOW(module, channel)
+ * @brief Allow the given module to print on the given channel
+ * @note If @see OSK_USE_RUNTIME_CONFIG is disabled then this macro doesn't do anything
+ * @param module is a @see osk_module
+ * @param channel is one of @see OSK_CHANNEL_INFO, @see OSK_CHANNEL_WARN, @see OSK_CHANNEL_ERROR,
+ * @see OSK_CHANNEL_ALL
+ * @return MALI_TRUE if the module is allowed to print on the channel.
+ */
+/**
+ * @def OSKP_PRINT_BLOCK(module, channel)
+ * @brief Prevent the given module from printing on the given channel
+ * @note If @see OSK_USE_RUNTIME_CONFIG is disabled then this macro doesn't do anything
+ * @param module is a @see osk_module
+ * @param channel is one of @see OSK_CHANNEL_INFO, @see OSK_CHANNEL_WARN, @see OSK_CHANNEL_ERROR,
+ * @see OSK_CHANNEL_ALL
+ * @return MALI_TRUE if the module is allowed to print on the channel.
+ */
+#if OSK_USE_RUNTIME_CONFIG
+ #define OSKP_PRINT_ALLOW(module, channel) oskp_debug_print_allow( (module), (channel) )
+ #define OSKP_PRINT_BLOCK(module, channel) oskp_debug_print_block( (module), (channel) )
+#else
+ #define OSKP_PRINT_ALLOW(module, channel) CSTD_NOP()
+ #define OSKP_PRINT_BLOCK(module, channel) CSTD_NOP()
+#endif
+
+/**
+ * @def OSKP_RAW_OUT(module, trace, ...)
+ * @brief (Private) system printing function associated to the @see OSK_PRINT_RAW event.
+ * @param module module ID
+ * @param trace location in the code from where the message is printed
+ * @param function function from where the message is printed
+ * @param ... Format string followed by format arguments.
+ */
+/* Select the correct system output function*/
+#if OSK_ON_RAW != OSK_ACTION_IGNORE
+ #define OSKP_RAW_OUT(module, trace, function, ...)\
+ do\
+ {\
+ OSKP_PRINT(__VA_ARGS__);\
+ OSKP_PRINT(OSK_STOP_MSG);\
+ }while(MALI_FALSE)
+#else
+ #define OSKP_RAW_OUT(module, trace, function, ...) CSTD_NOP()
+#endif
+
+
+/**
+ * @def OSKP_INFO_OUT(module, trace, ...)
+ * @brief (Private) system printing function associated to the @see OSK_PRINT_INFO event.
+ * @param module module ID
+ * @param trace location in the code from where the message is printed
+ * @param function function from where the message is printed
+ * @param ... Format string followed by format arguments.
+ */
+/* Select the correct system output function*/
+#if OSK_ON_INFO != OSK_ACTION_IGNORE
+ #define OSKP_INFO_OUT(module, trace, function, ...)\
+ do\
+ {\
+ /* Split up in several lines to prevent hitting max 128 chars limit of OSK print function */ \
+ OSKP_PRINT("Mali<INFO,%s>: ", module);\
+ OSKP_PRINT(__VA_ARGS__);\
+ OSKP_PRINT(OSK_STOP_MSG);\
+ }while(MALI_FALSE)
+#else
+ #define OSKP_INFO_OUT(module, trace, function, ...) CSTD_NOP()
+#endif
+
+/**
+ * @def OSKP_ASSERT_OUT(trace, function, ...)
+ * @brief (Private) system printing function associated to the @see OSKP_PRINT_ASSERT event.
+ * @param trace location in the code from where the message is printed
+ * @param function function from where the message is printed
+ * @param ... Format string followed by format arguments.
+ * @note function parameter cannot be concatenated with other strings
+ */
+/* Select the correct system output function*/
+#if OSK_ON_ASSERT != OSK_ACTION_IGNORE
+ #define OSKP_ASSERT_OUT(trace, function, ...)\
+ do\
+ {\
+ /* Split up in several lines to prevent hitting max 128 chars limit of OSK print function */ \
+ OSKP_PRINT("Mali<ASSERT>: %s function:%s ", trace, function);\
+ OSKP_PRINT(__VA_ARGS__);\
+ OSKP_PRINT(OSK_STOP_MSG);\
+ }while(MALI_FALSE)
+#else
+ #define OSKP_ASSERT_OUT(trace, function, ...) CSTD_NOP()
+#endif
+
+/**
+ * @def OSKP_WARN_OUT(module, trace, ...)
+ * @brief (Private) system printing function associated to the @see OSK_PRINT_WARN event.
+ * @param module module ID
+ * @param trace location in the code from where the message is printed
+ * @param function function from where the message is printed
+ * @param ... Format string followed by format arguments.
+ * @note function parameter cannot be concatenated with other strings
+ */
+/* Select the correct system output function*/
+#if OSK_ON_WARN != OSK_ACTION_IGNORE
+ #define OSKP_WARN_OUT(module, trace, function, ...)\
+ do\
+ {\
+ /* Split up in several lines to prevent hitting max 128 chars limit of OSK print function */ \
+ OSKP_PRINT("Mali<WARN,%s>: ", module);\
+ OSKP_PRINT(__VA_ARGS__);\
+ OSKP_PRINT(OSK_STOP_MSG);\
+ }while(MALI_FALSE)
+#else
+ #define OSKP_WARN_OUT(module, trace, function, ...) CSTD_NOP()
+#endif
+
+/**
+ * @def OSKP_ERROR_OUT(module, trace, ...)
+ * @brief (Private) system printing function associated to the @see OSK_PRINT_ERROR event.
+ * @param module module ID
+ * @param trace location in the code from where the message is printed
+ * @param function function from where the message is printed
+ * @param ... Format string followed by format arguments.
+ * @note function parameter cannot be concatenated with other strings
+ */
+/* Select the correct system output function*/
+#if OSK_ON_ERROR != OSK_ACTION_IGNORE
+ #define OSKP_ERROR_OUT(module, trace, function, ...)\
+ do\
+ {\
+ /* Split up in several lines to prevent hitting max 128 chars limit of OSK print function */ \
+ OSKP_PRINT("Mali<ERROR,%s>: ", module);\
+ OSKP_PRINT(__VA_ARGS__);\
+ OSKP_PRINT(OSK_STOP_MSG);\
+ }while(MALI_FALSE)
+#else
+ #define OSKP_ERROR_OUT(module, trace, function, ...) CSTD_NOP()
+#endif
+
+/**
+ * @def OSKP_PRINT_IS_ALLOWED(module, channel)
+ * @brief function or constant indicating if the given module is allowed to print on the given channel
+ * @note If @see OSK_USE_RUNTIME_CONFIG is disabled then this macro is set to MALI_TRUE to avoid any overhead
+ * @param module is a @see osk_module
+ * @param channel is one of @see OSK_CHANNEL_INFO, @see OSK_CHANNEL_WARN, @see OSK_CHANNEL_ERROR,
+ * @see OSK_CHANNEL_ALL
+ * @return MALI_TRUE if the module is allowed to print on the channel.
+ */
+#if OSK_USE_RUNTIME_CONFIG
+ #define OSKP_PRINT_IS_ALLOWED(module, channel) oskp_is_allowed_to_print( (module), (channel) )
+#else
+ #define OSKP_PRINT_IS_ALLOWED(module, channel) MALI_TRUE
+#endif
+
+/**
+ * @def OSKP_SIMULATE_FAILURE_IS_ENABLED(module, feature)
+ * @brief Macro that evaluates as true if the specified feature is enabled for the given module
+ * @note If @ref OSK_USE_RUNTIME_CONFIG is disabled then this macro always evaluates as true.
+ * @param[in] module is a @ref cdbg_module
+ * @param[in] channel is one of @see OSK_CHANNEL_INFO, @see OSK_CHANNEL_WARN, @see OSK_CHANNEL_ERROR,
+ * @return MALI_TRUE if the feature is enabled
+ */
+#if OSK_USE_RUNTIME_CONFIG
+#define OSKP_SIMULATE_FAILURE_IS_ENABLED(module, channel) oskp_is_allowed_to_simulate_failure( (module), (channel) )
+#else
+#define OSKP_SIMULATE_FAILURE_IS_ENABLED(module, channel) MALI_TRUE
+#endif
+
+OSK_STATIC_INLINE void osk_debug_get_thread_info( u32 *thread_id, u32 *cpu_nr )
+{
+ OSK_ASSERT( thread_id != NULL );
+ OSK_ASSERT( cpu_nr != NULL );
+
+ /* This implementation uses the PID as shown in ps listings.
+ * On 64-bit systems, this could narrow from signed 64-bit to unsigned 32-bit */
+ *thread_id = (u32)task_pid_nr(current);
+
+ /* On 64-bit systems, this could narrow from unsigned 64-bit to unsigned 32-bit */
+ *cpu_nr = (u32)task_cpu(current);
+}
+
+
+#endif /* _OSK_ARCH_DEBUG_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_LOCKS_H_
+#define _OSK_ARCH_LOCKS_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+/**
+ * Private macro to safely allow asserting on a mutex/rwlock/spinlock/irq
+ * spinlock pointer whilst still allowing its name to appear during
+ * CONFIG_PROVE_LOCKING.
+ *
+ * It's safe because \a lock can safely have side-effects.
+ *
+ * Makes use of a GNU C extension, but this macro is only needed under Linux
+ * anyway.
+ *
+ * NOTE: the local variable must not conflict with an identifier in a wider
+ * scope
+ *
+ * NOTE: Due to the way this is used in this file, this definition must persist
+ * outside of this file
+ */
+#define OSKP_LOCK_PTR_ASSERT( lock ) \
+ ({ \
+ __typeof__( lock ) __oskp_lock__ = ( lock ); \
+ OSK_ASSERT( NULL != __oskp_lock__ ); \
+ __oskp_lock__; })
+
+/*
+ * A definition must be provided of each lock init function to eliminate
+ * warnings. They'lll be hidden by subsequent macro definitions
+ */
+
+OSK_STATIC_INLINE osk_error osk_rwlock_init(osk_rwlock * const lock, osk_lock_order order)
+{
+ OSK_ASSERT_MSG( MALI_FALSE,
+ "FATAL: this definition of osk_rwlock_init() should've been uncallable - a macro redefines it\n" );
+ CSTD_UNUSED( lock );
+ CSTD_UNUSED( order );
+ return OSK_ERR_FAIL;
+}
+
+OSK_STATIC_INLINE osk_error osk_mutex_init(osk_mutex * const lock, osk_lock_order order)
+{
+ OSK_ASSERT_MSG( MALI_FALSE,
+ "FATAL: this definition of osk_mutex_init() should've been uncallable - a macro redefines it\n" );
+ CSTD_UNUSED( lock );
+ CSTD_UNUSED( order );
+ return OSK_ERR_FAIL;
+}
+
+OSK_STATIC_INLINE osk_error osk_spinlock_init(osk_spinlock * const lock, osk_lock_order order)
+{
+ OSK_ASSERT_MSG( MALI_FALSE,
+ "FATAL: this definition of osk_spinlock_init() should've been uncallable - a macro redefines it\n" );
+ CSTD_UNUSED( lock );
+ CSTD_UNUSED( order );
+ return OSK_ERR_FAIL;
+}
+
+OSK_STATIC_INLINE osk_error osk_spinlock_irq_init(osk_spinlock_irq * const lock, osk_lock_order order)
+{
+ OSK_ASSERT_MSG( MALI_FALSE,
+ "FATAL: this definition of osk_spinlock_irq_init() should've been uncallable - a macro redefines it\n" );
+ CSTD_UNUSED( lock );
+ CSTD_UNUSED( order );
+ return OSK_ERR_FAIL;
+}
+
+/*
+ * End of 'dummy' definitions
+ */
+
+
+/* Note: This uses a GNU C Extension to allow Linux's CONFIG_PROVE_LOCKING to work correctly
+ *
+ * This is not required outside of Linux
+ *
+ * NOTE: the local variable must not conflict with an identifier in a wider scope */
+#define osk_rwlock_init( ARG_LOCK, ARG_ORDER ) \
+ ({ \
+ osk_lock_order __oskp_order__ = (ARG_ORDER); \
+ OSK_ASSERT( OSK_LOCK_ORDER_LAST <= __oskp_order__ && __oskp_order__ <= OSK_LOCK_ORDER_FIRST ); \
+ init_rwsem( OSKP_LOCK_PTR_ASSERT((ARG_LOCK)) ); \
+ OSK_ERR_NONE;})
+
+OSK_STATIC_INLINE void osk_rwlock_term(osk_rwlock * lock)
+{
+ OSK_ASSERT(NULL != lock);
+ /* nop */
+}
+
+OSK_STATIC_INLINE void osk_rwlock_read_lock(osk_rwlock * lock)
+{
+ OSK_ASSERT(NULL != lock);
+ down_read(lock);
+}
+
+OSK_STATIC_INLINE void osk_rwlock_read_unlock(osk_rwlock * lock)
+{
+ OSK_ASSERT(NULL != lock);
+ up_read(lock);
+}
+
+OSK_STATIC_INLINE void osk_rwlock_write_lock(osk_rwlock * lock)
+{
+ OSK_ASSERT(NULL != lock);
+ down_write(lock);
+}
+
+OSK_STATIC_INLINE void osk_rwlock_write_unlock(osk_rwlock * lock)
+{
+ OSK_ASSERT(NULL != lock);
+ up_write(lock);
+}
+
+/* Note: This uses a GNU C Extension to allow Linux's CONFIG_PROVE_LOCKING to work correctly
+ *
+ * This is not required outside of Linux
+ *
+ * NOTE: the local variable must not conflict with an identifier in a wider scope */
+#define osk_mutex_init( ARG_LOCK, ARG_ORDER ) \
+ ({ \
+ osk_lock_order __oskp_order__ = (ARG_ORDER); \
+ OSK_ASSERT( OSK_LOCK_ORDER_LAST <= __oskp_order__ && __oskp_order__ <= OSK_LOCK_ORDER_FIRST ); \
+ mutex_init( OSKP_LOCK_PTR_ASSERT((ARG_LOCK)) ); \
+ OSK_ERR_NONE;})
+
+
+OSK_STATIC_INLINE void osk_mutex_term(osk_mutex * lock)
+{
+ OSK_ASSERT(NULL != lock);
+ return; /* nop */
+}
+
+OSK_STATIC_INLINE void osk_mutex_lock(osk_mutex * lock)
+{
+ OSK_ASSERT(NULL != lock);
+ mutex_lock(lock);
+}
+
+OSK_STATIC_INLINE void osk_mutex_unlock(osk_mutex * lock)
+{
+ OSK_ASSERT(NULL != lock);
+ mutex_unlock(lock);
+}
+
+/* Note: This uses a GNU C Extension to allow Linux's CONFIG_PROVE_LOCKING to work correctly
+ *
+ * This is not required outside of Linux
+ *
+ * NOTE: the local variable must not conflict with an identifier in a wider scope */
+#define osk_spinlock_init( ARG_LOCK, ARG_ORDER ) \
+ ({ \
+ osk_lock_order __oskp_order__ = (ARG_ORDER); \
+ OSK_ASSERT( OSK_LOCK_ORDER_LAST <= __oskp_order__ && __oskp_order__ <= OSK_LOCK_ORDER_FIRST ); \
+ spin_lock_init( OSKP_LOCK_PTR_ASSERT((ARG_LOCK)) ); \
+ OSK_ERR_NONE;})
+
+OSK_STATIC_INLINE void osk_spinlock_term(osk_spinlock * lock)
+{
+ OSK_ASSERT(NULL != lock);
+ /* nop */
+}
+
+OSK_STATIC_INLINE void osk_spinlock_lock(osk_spinlock * lock)
+{
+ OSK_ASSERT(NULL != lock);
+ spin_lock(lock);
+}
+
+OSK_STATIC_INLINE void osk_spinlock_unlock(osk_spinlock * lock)
+{
+ OSK_ASSERT(NULL != lock);
+ spin_unlock(lock);
+}
+
+/* Note: This uses a GNU C Extension to allow Linux's CONFIG_PROVE_LOCKING to work correctly
+ *
+ * This is not required outside of Linux
+ *
+ * NOTE: the local variable must not conflict with an identifier in a wider scope */
+#define osk_spinlock_irq_init( ARG_LOCK, ARG_ORDER ) \
+ ({ \
+ osk_lock_order __oskp_order__ = (ARG_ORDER); \
+ OSK_ASSERT( OSK_LOCK_ORDER_LAST <= __oskp_order__ && __oskp_order__ <= OSK_LOCK_ORDER_FIRST ); \
+ spin_lock_init( &(OSKP_LOCK_PTR_ASSERT((ARG_LOCK))->lock) ); \
+ OSK_ERR_NONE;})
+
+OSK_STATIC_INLINE void osk_spinlock_irq_term(osk_spinlock_irq * lock)
+{
+ OSK_ASSERT(NULL != lock);
+}
+
+OSK_STATIC_INLINE void osk_spinlock_irq_lock(osk_spinlock_irq * lock)
+{
+ OSK_ASSERT(NULL != lock);
+ spin_lock_irqsave(&lock->lock, lock->flags);
+}
+
+OSK_STATIC_INLINE void osk_spinlock_irq_unlock(osk_spinlock_irq * lock)
+{
+ OSK_ASSERT(NULL != lock);
+ spin_unlock_irqrestore(&lock->lock, lock->flags);
+}
+
+#endif /* _OSK_ARCH_LOCKS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_LOW_LEVEL_MEM_H_
+#define _OSK_ARCH_LOW_LEVEL_MEM_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#include <linux/mm.h>
+#include <linux/highmem.h>
+#include <linux/dma-mapping.h>
+
+struct oskp_phy_os_allocator
+{
+};
+
+OSK_STATIC_INLINE osk_error oskp_phy_os_allocator_init(oskp_phy_os_allocator * const allocator,
+ osk_phy_addr mem, u32 nr_pages)
+{
+ OSK_ASSERT(NULL != allocator);
+
+ return OSK_ERR_NONE;
+}
+
+OSK_STATIC_INLINE void oskp_phy_os_allocator_term(oskp_phy_os_allocator *allocator)
+{
+ OSK_ASSERT(NULL != allocator);
+ /* Nothing needed */
+}
+
+OSK_STATIC_INLINE u32 oskp_phy_os_pages_alloc(oskp_phy_os_allocator *allocator,
+ u32 nr_pages, osk_phy_addr *pages)
+{
+ int i;
+
+ OSK_ASSERT(NULL != allocator);
+
+ if(OSK_SIMULATE_FAILURE(OSK_OSK))
+ {
+ return 0;
+ }
+
+ for (i = 0; i < nr_pages; i++)
+ {
+ struct page *p;
+ void * mp;
+
+ p = alloc_page(__GFP_IO |
+ __GFP_FS |
+ __GFP_COLD |
+ __GFP_NOWARN |
+ __GFP_NORETRY |
+ __GFP_NOMEMALLOC |
+ __GFP_HIGHMEM |
+ __GFP_HARDWALL);
+ if (NULL == p)
+ {
+ break;
+ }
+
+ mp = kmap(p);
+ if (NULL == mp)
+ {
+ __free_page(p);
+ break;
+ }
+
+ memset(mp, 0x00, PAGE_SIZE); /* instead of __GFP_ZERO, so we can do cache maintenance */
+ osk_sync_to_memory(PFN_PHYS(page_to_pfn(p)), mp, PAGE_SIZE);
+ kunmap(p);
+
+ pages[i] = PFN_PHYS(page_to_pfn(p));
+ }
+
+ return i;
+}
+
+static inline void oskp_phy_os_pages_free(oskp_phy_os_allocator *allocator,
+ u32 nr_pages, osk_phy_addr *pages)
+{
+ int i;
+
+ OSK_ASSERT(NULL != allocator);
+
+ for (i = 0; i < nr_pages; i++)
+ {
+ if (0 != pages[i])
+ {
+ __free_page(pfn_to_page(PFN_DOWN(pages[i])));
+ pages[i] = (osk_phy_addr)0;
+ }
+ }
+}
+
+
+OSK_STATIC_INLINE osk_error oskp_phy_dedicated_allocator_request_memory(osk_phy_addr mem,u32 nr_pages, const char* name)
+{
+ if(OSK_SIMULATE_FAILURE(OSK_OSK))
+ {
+ return OSK_ERR_FAIL;
+ }
+
+ if (NULL != request_mem_region(mem, nr_pages << OSK_PAGE_SHIFT , name))
+ {
+ return OSK_ERR_NONE;
+ }
+ return OSK_ERR_FAIL;
+}
+
+OSK_STATIC_INLINE void oskp_phy_dedicated_allocator_release_memory(osk_phy_addr mem,u32 nr_pages)
+{
+ release_mem_region(mem, nr_pages << OSK_PAGE_SHIFT);
+}
+
+
+static inline void *osk_kmap(osk_phy_addr page)
+{
+ if(OSK_SIMULATE_FAILURE(OSK_OSK))
+ {
+ return NULL;
+ }
+
+ return kmap(pfn_to_page(PFN_DOWN(page)));
+}
+
+static inline void osk_kunmap(osk_phy_addr page, void * mapping)
+{
+ kunmap(pfn_to_page(PFN_DOWN(page)));
+}
+
+static inline void *osk_kmap_atomic(osk_phy_addr page)
+{
+ /**
+ * Note: kmap_atomic should never fail and so OSK_SIMULATE_FAILURE is not
+ * included for this function call.
+ */
+ return kmap_atomic(pfn_to_page(PFN_DOWN(page)));
+}
+
+static inline void osk_kunmap_atomic(osk_phy_addr page, void *mapping)
+{
+ kunmap_atomic(mapping);
+}
+
+static inline void osk_sync_to_memory(osk_phy_addr paddr, osk_virt_addr vaddr, size_t sz)
+{
+#ifdef CONFIG_ARM
+ dmac_flush_range(vaddr, vaddr+sz-1);
+ outer_flush_range(paddr, paddr+sz-1);
+#elif defined(CONFIG_X86)
+ struct scatterlist scl = {0, };
+ sg_set_page(&scl, pfn_to_page(PFN_DOWN(paddr)), sz,
+ paddr & (PAGE_SIZE -1 ));
+ dma_sync_sg_for_cpu(NULL, &scl, 1, DMA_TO_DEVICE);
+ mb(); /* for outer_sync (if needed) */
+#else
+#error Implement cache maintenance for your architecture here
+#endif
+}
+
+static inline void osk_sync_to_cpu(osk_phy_addr paddr, osk_virt_addr vaddr, size_t sz)
+{
+#ifdef CONFIG_ARM
+ dmac_flush_range(vaddr, vaddr+sz-1);
+ outer_flush_range(paddr, paddr+sz-1);
+#elif defined(CONFIG_X86)
+ struct scatterlist scl = {0, };
+ sg_set_page(&scl, pfn_to_page(PFN_DOWN(paddr)), sz,
+ paddr & (PAGE_SIZE -1 ));
+ dma_sync_sg_for_cpu(NULL, &scl, 1, DMA_FROM_DEVICE);
+#else
+#error Implement cache maintenance for your architecture here
+#endif
+}
+
+#endif /* _OSK_ARCH_LOW_LEVEL_MEM_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_MATH_H
+#define _OSK_ARCH_MATH_H
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#include <asm/div64.h>
+
+OSK_STATIC_INLINE u32 osk_divmod6432(u64 *value, u32 divisor)
+{
+ u64 v;
+ u32 r;
+
+ OSK_ASSERT(NULL != value);
+
+ v = *value;
+ r = do_div(v, divisor);
+ *value = v;
+ return r;
+}
+
+#endif /* _OSK_ARCH_MATH_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2008-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_MEM_H_
+#define _OSK_ARCH_MEM_H_
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+
+OSK_STATIC_INLINE void * osk_malloc(size_t size)
+{
+ OSK_ASSERT(0 != size);
+
+ if(OSK_SIMULATE_FAILURE(OSK_OSK))
+ {
+ return NULL;
+ }
+
+ return kmalloc(size, GFP_KERNEL);
+}
+
+OSK_STATIC_INLINE void * osk_calloc(size_t size)
+{
+ OSK_ASSERT(0 != size);
+
+ if(OSK_SIMULATE_FAILURE(OSK_OSK))
+ {
+ return NULL;
+ }
+
+ return kzalloc(size, GFP_KERNEL);
+}
+
+OSK_STATIC_INLINE void osk_free(void * ptr)
+{
+ kfree(ptr);
+}
+
+OSK_STATIC_INLINE void * osk_vmalloc(size_t size)
+{
+ OSK_ASSERT(0 != size);
+
+ if(OSK_SIMULATE_FAILURE(OSK_OSK))
+ {
+ return NULL;
+ }
+
+ return vmalloc_user(size);
+}
+
+OSK_STATIC_INLINE void osk_vfree(void * ptr)
+{
+ vfree(ptr);
+}
+
+#define OSK_MEMCPY( dst, src, len ) memcpy(dst, src, len)
+
+#define OSK_MEMCMP( s1, s2, len ) memcmp(s1, s2, len)
+
+#define OSK_MEMSET( ptr, chr, size ) memset(ptr, chr, size)
+
+
+#endif /* _OSK_ARCH_MEM_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_POWER_H_
+#define _OSK_ARCH_POWER_H_
+
+#include <linux/pm_runtime.h>
+
+OSK_STATIC_INLINE osk_power_request_result osk_power_request(osk_power_info *info, osk_power_state state)
+{
+ osk_power_request_result request_result = OSK_POWER_REQUEST_FINISHED;
+
+ OSK_ASSERT(NULL != info);
+
+ switch(state)
+ {
+ case OSK_POWER_STATE_OFF:
+ /* request OS to suspend device*/
+ break;
+ case OSK_POWER_STATE_IDLE:
+ /* request OS to idle device */
+ break;
+ case OSK_POWER_STATE_ACTIVE:
+ /* request OS to resume device */
+ break;
+ }
+ return request_result;
+}
+
+#endif
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_TIME_H
+#define _OSK_ARCH_TIME_H
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+OSK_STATIC_INLINE osk_ticks osk_time_now(void)
+{
+ return jiffies;
+}
+
+OSK_STATIC_INLINE u32 osk_time_mstoticks(u32 ms)
+{
+ return msecs_to_jiffies(ms);
+}
+
+OSK_STATIC_INLINE u32 osk_time_elapsed(osk_ticks ticka, osk_ticks tickb)
+{
+ return jiffies_to_msecs((long)tickb - (long)ticka);
+}
+
+OSK_STATIC_INLINE mali_bool osk_time_after(osk_ticks ticka, osk_ticks tickb)
+{
+ return time_after(ticka, tickb);
+}
+
+OSK_STATIC_INLINE void osk_gettimeofday(osk_timeval *tv)
+{
+ struct timespec ts;
+ getnstimeofday(&ts);
+
+ tv->tv_sec = ts.tv_sec;
+ tv->tv_usec = ts.tv_nsec/1000;
+}
+
+#endif /* _OSK_ARCH_TIME_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_arch_timers.h
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_TIMERS_H
+#define _OSK_ARCH_TIMERS_H
+
+#if MALI_LICENSE_IS_GPL
+ #include "mali_osk_arch_timers_gpl.h"
+#else
+ #include "mali_osk_arch_timers_commercial.h"
+#endif
+
+#endif /* _OSK_ARCH_TIMERS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_arch_timers_commercial.h
+ * Implementation of the OS abstraction layer for the kernel device driver
+ * For Commercial builds - does not use the hrtimers functionality.
+ */
+
+#ifndef _OSK_ARCH_TIMERS_COMMERCIAL_H
+#define _OSK_ARCH_TIMERS_COMMERCIAL_H
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+OSK_STATIC_INLINE osk_error osk_timer_init(osk_timer * const tim)
+{
+ OSK_ASSERT(NULL != tim);
+ init_timer(&tim->timer);
+ OSK_DEBUG_CODE( tim->active = MALI_FALSE );
+ OSK_ASSERT(0 == object_is_on_stack(tim));
+ return OSK_ERR_NONE;
+}
+
+OSK_STATIC_INLINE osk_error osk_timer_on_stack_init(osk_timer * const tim)
+{
+ OSK_ASSERT(NULL != tim);
+ init_timer_on_stack(&tim->timer);
+ OSK_DEBUG_CODE( tim->active = MALI_FALSE );
+ OSK_ASSERT(0 != object_is_on_stack(tim));
+ return OSK_ERR_NONE;
+}
+
+OSK_STATIC_INLINE osk_error osk_timer_start(osk_timer *tim, u32 delay)
+{
+ OSK_ASSERT(NULL != tim);
+ OSK_ASSERT(NULL != tim->timer.function);
+ OSK_ASSERT(0 != delay);
+ tim->timer.expires = jiffies + ((delay * HZ + 999) / 1000);
+ add_timer(&tim->timer);
+ OSK_DEBUG_CODE( tim->active = MALI_TRUE );
+ return OSK_ERR_NONE;
+}
+
+OSK_STATIC_INLINE osk_error osk_timer_start_ns(osk_timer *tim, u64 delay)
+{
+ osk_error err;
+ osk_divmod6432(&delay, 1000000);
+
+ err = osk_timer_start( tim, delay );
+
+ return err;
+}
+
+OSK_STATIC_INLINE osk_error osk_timer_modify(osk_timer *tim, u32 delay)
+{
+ OSK_ASSERT(NULL != tim);
+ OSK_ASSERT(NULL != tim->timer.function);
+ OSK_ASSERT(0 != delay);
+ mod_timer(&tim->timer, jiffies + ((delay * HZ + 999) / 1000));
+ OSK_DEBUG_CODE( tim->active = MALI_TRUE );
+ return OSK_ERR_NONE;
+}
+
+OSK_STATIC_INLINE osk_error osk_timer_modify_ns(osk_timer *tim, u64 new_delay)
+{
+ osk_error err;
+ osk_divmod6432(&new_delay, 1000000);
+
+ err = osk_timer_modify( tim, new_delay );
+ return err;
+}
+
+OSK_STATIC_INLINE void osk_timer_stop(osk_timer *tim)
+{
+ OSK_ASSERT(NULL != tim);
+ OSK_ASSERT(NULL != tim->timer.function);
+ del_timer_sync(&tim->timer);
+ OSK_DEBUG_CODE( tim->active = MALI_FALSE );
+}
+
+OSK_STATIC_INLINE void osk_timer_callback_set(osk_timer *tim, osk_timer_callback callback, void *data)
+{
+ OSK_ASSERT(NULL != tim);
+ OSK_ASSERT(NULL != callback);
+ OSK_DEBUG_CODE(
+ if (MALI_FALSE == tim->active)
+ {
+ }
+ );
+ /* osk_timer_callback uses void * for the callback parameter instead of unsigned long in Linux */
+ tim->timer.function = (void (*)(unsigned long))callback;
+ tim->timer.data = (unsigned long)data;
+}
+
+OSK_STATIC_INLINE void osk_timer_term(osk_timer *tim)
+{
+ OSK_ASSERT(NULL != tim);
+ OSK_ASSERT(0 == object_is_on_stack(tim));
+ OSK_DEBUG_CODE(
+ if (MALI_FALSE == tim->active)
+ {
+ }
+ );
+ /* Nothing to do */
+}
+
+OSK_STATIC_INLINE void osk_timer_on_stack_term(osk_timer *tim)
+{
+ OSK_ASSERT(NULL != tim);
+ OSK_ASSERT(0 != object_is_on_stack(tim));
+ OSK_DEBUG_CODE(
+ if (MALI_FALSE == tim->active)
+ {
+ }
+ );
+ destroy_timer_on_stack(&tim->timer);
+}
+
+#endif /* _OSK_ARCH_TIMERS_COMMERCIAL_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_osk_arch_timers_gpl.h
+ * Implementation of the OS abstraction layer for the kernel device driver
+ * For GPL builds - uses the hrtimers functionality.
+ */
+
+#ifndef _OSK_ARCH_TIMERS_GPL_H
+#define _OSK_ARCH_TIMERS_GPL_H
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#if MALI_DEBUG != 0
+void oskp_debug_test_timer_stats( void );
+#endif
+
+enum hrtimer_restart oskp_timer_callback_wrapper( struct hrtimer * hr_timer );
+
+OSK_STATIC_INLINE osk_error osk_timer_init(osk_timer * const tim)
+{
+ OSK_ASSERT(NULL != tim);
+
+ OSK_DEBUG_CODE( oskp_debug_test_timer_stats() );
+
+ hrtimer_init(&tim->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+ tim->timer.function = NULL;
+
+ OSK_DEBUG_CODE( tim->active = MALI_FALSE );
+ OSK_ASSERT(0 == object_is_on_stack(tim));
+ return OSK_ERR_NONE;
+}
+
+OSK_STATIC_INLINE osk_error osk_timer_on_stack_init(osk_timer * const tim)
+{
+ OSK_ASSERT(NULL != tim);
+ hrtimer_init_on_stack(&tim->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+ tim->timer.function = NULL;
+
+ OSK_DEBUG_CODE( tim->active = MALI_FALSE );
+ OSK_ASSERT(0 != object_is_on_stack(tim));
+ return OSK_ERR_NONE;
+}
+
+OSK_STATIC_INLINE osk_error osk_timer_start(osk_timer *tim, u32 delay)
+{
+ osk_error err;
+ u64 delay_ns = delay * (u64)1000000U;
+
+ err = osk_timer_start_ns( tim, delay_ns );
+
+ return err;
+}
+
+OSK_STATIC_INLINE osk_error osk_timer_start_ns(osk_timer *tim, u64 delay_ns)
+{
+ ktime_t kdelay;
+ int was_active;
+ OSK_ASSERT(NULL != tim);
+ OSK_ASSERT(NULL != tim->timer.function);
+ OSK_ASSERT(delay_ns != 0);
+
+ kdelay = ns_to_ktime( delay_ns );
+
+ was_active = hrtimer_start( &tim->timer, kdelay, HRTIMER_MODE_REL );
+
+ OSK_ASSERT( was_active == 0 ); /* You cannot start a timer that has already been started */
+
+ CSTD_UNUSED( was_active );
+ OSK_DEBUG_CODE( tim->active = MALI_TRUE );
+ return OSK_ERR_NONE;
+}
+
+OSK_STATIC_INLINE osk_error osk_timer_modify(osk_timer *tim, u32 new_delay)
+{
+ osk_error err;
+ u64 delay_ns = new_delay * (u64)1000000U;
+
+ err = osk_timer_modify_ns( tim, delay_ns );
+ return err;
+}
+
+OSK_STATIC_INLINE osk_error osk_timer_modify_ns(osk_timer *tim, u64 new_delay_ns)
+{
+ ktime_t kdelay;
+ int was_active;
+ OSK_ASSERT(NULL != tim);
+ OSK_ASSERT(NULL != tim->timer.function);
+ OSK_ASSERT(0 != new_delay_ns);
+
+ kdelay = ns_to_ktime( new_delay_ns );
+
+ /* hrtimers will stop the existing timer if it's running on any cpu, so
+ * it's safe just to start the timer again: */
+ was_active = hrtimer_start( &tim->timer, kdelay, HRTIMER_MODE_REL );
+
+ CSTD_UNUSED( was_active );
+ OSK_DEBUG_CODE( tim->active = MALI_TRUE );
+ return OSK_ERR_NONE;
+}
+
+OSK_STATIC_INLINE void osk_timer_stop(osk_timer *tim)
+{
+ int was_active;
+ OSK_ASSERT(NULL != tim);
+ OSK_ASSERT(NULL != tim->timer.function);
+
+ was_active = hrtimer_cancel(&tim->timer);
+
+ CSTD_UNUSED( was_active );
+ OSK_DEBUG_CODE( tim->active = MALI_FALSE );
+}
+
+OSK_STATIC_INLINE void osk_timer_callback_set(osk_timer *tim, osk_timer_callback callback, void *data)
+{
+ OSK_ASSERT(NULL != tim);
+ OSK_ASSERT(NULL != callback);
+ OSK_DEBUG_CODE(
+ if (MALI_FALSE == tim->active)
+ {
+ }
+ );
+
+ tim->timer.function = &oskp_timer_callback_wrapper;
+
+ /* osk_timer_callback uses void * for the callback parameter instead of unsigned long in Linux */
+ tim->callback = callback;
+ tim->data = data;
+}
+
+OSK_STATIC_INLINE void osk_timer_term(osk_timer *tim)
+{
+ OSK_ASSERT(NULL != tim);
+ OSK_ASSERT(0 == object_is_on_stack(tim));
+ OSK_DEBUG_CODE(
+ if (MALI_FALSE == tim->active)
+ {
+ }
+ );
+ /* Nothing to do */
+}
+
+OSK_STATIC_INLINE void osk_timer_on_stack_term(osk_timer *tim)
+{
+ OSK_ASSERT(NULL != tim);
+ OSK_ASSERT(0 != object_is_on_stack(tim));
+ OSK_DEBUG_CODE(
+ if (MALI_FALSE == tim->active)
+ {
+ }
+ );
+ destroy_hrtimer_on_stack(&tim->timer);
+}
+
+#endif /* _OSK_ARCH_TIMERS_GPL_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_TYPES_H_
+#define _OSK_ARCH_TYPES_H_
+
+#include <linux/version.h> /* version detection */
+#include <linux/spinlock.h>
+#include <linux/mutex.h>
+#include <linux/rwsem.h>
+#include <linux/wait.h>
+#if MALI_LICENSE_IS_GPL
+#include <linux/hrtimer.h>
+#endif
+#include <linux/workqueue.h>
+#include <linux/mm_types.h>
+#include <asm/atomic.h>
+#include <linux/sched.h>
+
+#include <malisw/mali_malisw.h>
+
+ /* This will have to agree with the OSU definition of the CPU page size: CONFIG_CPU_PAGE_SIZE_LOG2 */
+#define OSK_PAGE_SHIFT PAGE_SHIFT
+#define OSK_PAGE_SIZE PAGE_SIZE
+#define OSK_PAGE_MASK PAGE_MASK
+
+/** Number of CPU Cores */
+#define OSK_NUM_CPUS NR_CPUS
+
+/** Total amount of memory, in pages */
+#define OSK_MEM_PAGES totalram_pages
+
+/**
+ * @def OSK_L1_DCACHE_LINE_SIZE_LOG2
+ * @brief CPU L1 Data Cache Line size, in the form of a Logarithm to base 2.
+ *
+ * Must agree with the OSU definition: CONFIG_CPU_L1_DCACHE_LINE_SIZE_LOG2.
+ */
+#define OSK_L1_DCACHE_LINE_SIZE_LOG2 6
+
+/**
+ * @def OSK_L1_DCACHE_SIZE
+ * @brief CPU L1 Data Cache size, in bytes.
+ *
+ * Must agree with the OSU definition: CONFIG_CPU_L1_DCACHE_SIZE.
+ */
+#define OSK_L1_DCACHE_SIZE ((u32)0x00008000)
+
+
+#define OSK_MIN(x,y) min((x), (y))
+
+typedef spinlock_t osk_spinlock;
+typedef struct osk_spinlock_irq {
+ spinlock_t lock;
+ unsigned long flags;
+} osk_spinlock_irq;
+
+typedef struct mutex osk_mutex;
+typedef struct rw_semaphore osk_rwlock;
+
+typedef atomic_t osk_atomic;
+
+typedef struct osk_waitq
+{
+ /**
+ * set to MALI_TRUE when the waitq is signaled; set to MALI_FALSE when
+ * not signaled.
+ *
+ * This does not require locking for the setter, clearer and waiter.
+ * Here's why:
+ * - The only sensible use of a waitq is for operations to occur in
+ * strict order, without possibility of race between the callers of
+ * osk_waitq_set() and osk_waitq_clear() (the setter and clearer).
+ * Effectively, the clear must cause a later set to occur.
+ * - When the clear/set operations occur in different threads, some
+ * form of communication needs to happen for the clear to cause the
+ * signal to occur later.
+ * - This itself \b must involve a memory barrier, and so the clear is
+ * guarenteed to be observed by the waiter such that it is before the set.
+ * (and the set is observed after the clear).
+ *
+ * For example, running a GPU job might clear a "there are jobs in
+ * flight" waitq. Running the job must issue an register write, (and
+ * likely a post to a workqueue due to IRQ handling). Those operations
+ * must cause a data barrier to occur. During IRQ handling/workqueue
+ * processing, we might then set the waitq, and this happens after the
+ * barrier. Hence, the set and clear are observed in strict order.
+ */
+ mali_bool signaled;
+ wait_queue_head_t wq; /**< threads waiting for flag to be signalled */
+} osk_waitq;
+
+typedef struct osk_timer {
+#if MALI_LICENSE_IS_GPL
+ struct hrtimer timer;
+ osk_timer_callback callback;
+ void *data;
+#else /* MALI_LICENSE_IS_GPL */
+ struct timer_list timer;
+#endif /* MALI_LICENSE_IS_GPL */
+#ifdef MALI_DEBUG
+ mali_bool active;
+#endif
+} osk_timer;
+
+typedef struct page osk_page;
+typedef struct vm_area_struct osk_vma;
+
+typedef unsigned long osk_ticks; /* 32-bit resolution deemed to be sufficient */
+
+/* Separate definitions for the following, to avoid wrapper functions for GPL drivers */
+#if MALI_LICENSE_IS_GPL
+typedef work_func_t osk_workq_fn;
+
+typedef struct work_struct osk_workq_work;
+
+typedef struct osk_workq
+{
+ struct workqueue_struct *wqs;
+} osk_workq;
+
+#else /* MALI_LICENSE_IS_GPL */
+
+/* Forward decls */
+typedef struct osk_workq_work osk_workq_work;
+typedef struct osk_workq osk_workq;
+
+typedef void (*osk_workq_fn)(osk_workq_work *);
+
+struct osk_workq_work
+{
+ struct work_struct os_work;
+ /* Non-GPL driver must manually track work */
+ osk_workq_fn actual_fn;
+ osk_workq *parent_wq;
+};
+
+struct osk_workq
+{
+ /* Non-GPL driver shouldn't flush the global workqueue, so we do a manual form of flushing */
+ spinlock_t active_items_lock;
+ u32 nr_active_items;
+ wait_queue_head_t waitq_zero_active_items;
+};
+
+#endif /* MALI_LICENSE_IS_GPL */
+
+typedef struct device osk_power_info;
+
+typedef struct timeval osk_timeval;
+
+/**
+ * Physical address
+ */
+typedef phys_addr_t osk_phy_addr;
+
+#endif /* _OSK_ARCH_TYPES_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_WAITQ_H
+#define _OSK_ARCH_WAITQ_H
+
+#ifndef _OSK_H_
+#error "Include mali_osk.h directly"
+#endif
+
+#include <linux/wait.h>
+#include <linux/sched.h>
+
+/*
+ * Note:
+ *
+ * We do not need locking on the signalled member (see its doxygen description)
+ */
+
+OSK_STATIC_INLINE osk_error osk_waitq_init(osk_waitq * const waitq)
+{
+ OSK_ASSERT(NULL != waitq);
+ waitq->signaled = MALI_FALSE;
+ init_waitqueue_head(&waitq->wq);
+ return OSK_ERR_NONE;
+}
+
+OSK_STATIC_INLINE void osk_waitq_wait(osk_waitq *waitq)
+{
+ OSK_ASSERT(NULL != waitq);
+ wait_event(waitq->wq, waitq->signaled != MALI_FALSE);
+}
+
+OSK_STATIC_INLINE void osk_waitq_set(osk_waitq *waitq)
+{
+ OSK_ASSERT(NULL != waitq);
+ waitq->signaled = MALI_TRUE;
+ wake_up(&waitq->wq);
+}
+
+OSK_STATIC_INLINE void osk_waitq_clear(osk_waitq *waitq)
+{
+ OSK_ASSERT(NULL != waitq);
+ waitq->signaled = MALI_FALSE;
+}
+
+OSK_STATIC_INLINE void osk_waitq_term(osk_waitq *waitq)
+{
+ OSK_ASSERT(NULL != waitq);
+ /* NOP on Linux */
+}
+
+#endif /* _OSK_ARCH_WAITQ_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file
+ * Implementation of the OS abstraction layer for the kernel device driver
+ */
+
+#ifndef _OSK_ARCH_WORKQ_H_
+#define _OSK_ARCH_WORKQ_H_
+
+#include <linux/version.h> /* version detection */
+#include "osk/include/mali_osk_failure.h"
+
+#if MALI_LICENSE_IS_GPL == 0
+/* Wrapper function to allow flushing in non-GPL driver */
+void oskp_work_func_wrapper( struct work_struct *work );
+
+/* Forward decl */
+OSK_STATIC_INLINE void osk_workq_flush(osk_workq *wq);
+#endif /* MALI_LICENSE_IS_GPL == 0 */
+
+OSK_STATIC_INLINE osk_error osk_workq_init(osk_workq * const wq, const char *name, u32 flags)
+{
+#if (MALI_LICENSE_IS_GPL) && (LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,36))
+ unsigned int wqflags = 0;
+#endif /*(MALI_LICENSE_IS_GPL) && (LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,36))*/
+
+ OSK_ASSERT(NULL != wq);
+ OSK_ASSERT(NULL != name);
+ OSK_ASSERT(0 == (flags & ~(OSK_WORKQ_NON_REENTRANT|OSK_WORKQ_HIGH_PRIORITY|OSK_WORKQ_RESCUER)));
+
+#if MALI_LICENSE_IS_GPL
+ if(OSK_SIMULATE_FAILURE(OSK_OSK))
+ {
+ return OSK_ERR_FAIL;
+ }
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,36)
+ if (flags & OSK_WORKQ_NON_REENTRANT)
+ {
+ wqflags |= WQ_NON_REENTRANT;
+ }
+ if (flags & OSK_WORKQ_HIGH_PRIORITY)
+ {
+ wqflags |= WQ_HIGHPRI;
+ }
+ if (flags & OSK_WORKQ_RESCUER)
+ {
+ wqflags |= WQ_RESCUER;
+ }
+
+ wq->wqs = alloc_workqueue(name, wqflags, 1);
+#else
+ if (flags & OSK_WORKQ_NON_REENTRANT)
+ {
+ wq->wqs = create_singlethread_workqueue(name);
+ }
+ else
+ {
+ wq->wqs = create_workqueue(name);
+ }
+#endif
+ if (NULL == wq->wqs)
+ {
+ return OSK_ERR_FAIL;
+ }
+#else
+ /* Non-GPL driver uses global workqueue */
+ spin_lock_init( &wq->active_items_lock );
+ wq->nr_active_items = 0;
+ init_waitqueue_head( &wq->waitq_zero_active_items );
+#endif
+ return OSK_ERR_NONE;
+}
+
+OSK_STATIC_INLINE void osk_workq_term(osk_workq *wq)
+{
+ OSK_ASSERT(NULL != wq);
+#if MALI_LICENSE_IS_GPL
+ destroy_workqueue(wq->wqs);
+#else
+ /* Non-GPL driver uses global workqueue, so flush it manually */
+ osk_workq_flush(wq);
+#endif
+}
+
+OSK_STATIC_INLINE void osk_workq_work_init(osk_workq_work * const wk, osk_workq_fn fn)
+{
+ work_func_t func_to_call_first;
+ struct work_struct *os_work;
+ OSK_ASSERT(NULL != wk);
+ OSK_ASSERT(NULL != fn);
+ OSK_ASSERT(0 == object_is_on_stack(wk));
+
+#if MALI_LICENSE_IS_GPL
+ func_to_call_first = fn;
+ os_work = wk;
+#else /* MALI_LICENSE_IS_GPL */
+ func_to_call_first = &oskp_work_func_wrapper;
+ wk->actual_fn = fn;
+ os_work = &wk->os_work;
+#endif /* MALI_LICENSE_IS_GPL */
+
+ INIT_WORK(os_work, func_to_call_first);
+}
+
+OSK_STATIC_INLINE void osk_workq_work_init_on_stack(osk_workq_work * const wk, osk_workq_fn fn)
+{
+ work_func_t func_to_call_first;
+ struct work_struct *os_work;
+ OSK_ASSERT(NULL != wk);
+ OSK_ASSERT(NULL != fn);
+ OSK_ASSERT(0 != object_is_on_stack(wk));
+
+#if MALI_LICENSE_IS_GPL
+ func_to_call_first = fn;
+ os_work = wk;
+#else /* MALI_LICENSE_IS_GPL */
+ func_to_call_first = &oskp_work_func_wrapper;
+ wk->actual_fn = fn;
+ os_work = &wk->os_work;
+#endif /* MALI_LICENSE_IS_GPL */
+
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,37)
+ INIT_WORK_ONSTACK(os_work, func_to_call_first);
+#else
+ INIT_WORK_ON_STACK(os_work, func_to_call_first);
+#endif
+}
+
+OSK_STATIC_INLINE void osk_workq_submit(osk_workq *wq, osk_workq_work * const wk)
+{
+ OSK_ASSERT(NULL != wk);
+ OSK_ASSERT(NULL != wq);
+
+#if MALI_LICENSE_IS_GPL
+ queue_work(wq->wqs, wk);
+#else
+ /* Non-GPL driver uses global workqueue */
+ {
+ unsigned long flags;
+
+ wk->parent_wq = wq;
+
+ spin_lock_irqsave( &wq->active_items_lock, flags );
+ ++(wq->nr_active_items);
+ spin_unlock_irqrestore( &wq->active_items_lock, flags );
+ schedule_work(&wk->os_work);
+ }
+#endif
+}
+
+OSK_STATIC_INLINE void osk_workq_flush(osk_workq *wq)
+{
+ OSK_ASSERT(NULL != wq);
+
+#if MALI_LICENSE_IS_GPL
+ flush_workqueue(wq->wqs);
+#else
+ /* Non-GPL driver uses global workqueue, but we shouldn't use flush_scheduled_work() */
+
+ /* Locking is required here, to ensure that the wait_queue object doesn't
+ * disappear from the work item. This is the sequence we must guard
+ * against:
+ * - work item thread (signaller) atomic-sets nr_active_items = 0
+ * - signaller gets scheduled out
+ * - another thread (waiter) calls osk_workq_term()
+ * - ...which calls osk_workq_flush()
+ * - ...which calls wait_event(), and immediately returns without blocking
+ * - osk_workq_term() completes
+ * - caller (waiter) frees the memory backing the osk_workq
+ * - signaller gets scheduled back in
+ * - tries to call wake_up() on a freed address - error!
+ */
+ {
+ unsigned long flags;
+ spin_lock_irqsave( &wq->active_items_lock, flags );
+ while( wq->nr_active_items != 0 )
+ {
+ spin_unlock_irqrestore( &wq->active_items_lock, flags );
+ wait_event( wq->waitq_zero_active_items, (wq->nr_active_items == 0) );
+ spin_lock_irqsave( &wq->active_items_lock, flags );
+ }
+ spin_unlock_irqrestore( &wq->active_items_lock, flags );
+ }
+#endif
+}
+
+
+#endif
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#include <osk/mali_osk.h>
+
+oskp_debug_assert_cb oskp_debug_assert_registered_cb =
+{
+ NULL,
+ NULL
+};
+
+void oskp_debug_print(const char *format, ...)
+{
+ va_list args;
+ va_start(args, format);
+ oskp_validate_format_string(format);
+ vprintk(format, args);
+ va_end(args);
+}
+
+s32 osk_snprintf(char *str, size_t size, const char *format, ...)
+{
+ va_list args;
+ s32 ret;
+ va_start(args, format);
+ oskp_validate_format_string(format);
+ ret = vsnprintf(str, size, format, args);
+ va_end(args);
+ return ret;
+}
+
+void osk_debug_assert_register_hook( osk_debug_assert_hook *func, void *param )
+{
+ oskp_debug_assert_registered_cb.func = func;
+ oskp_debug_assert_registered_cb.param = param;
+}
+
+void oskp_debug_assert_call_hook( void )
+{
+ if ( oskp_debug_assert_registered_cb.func != NULL )
+ {
+ oskp_debug_assert_registered_cb.func( oskp_debug_assert_registered_cb.param );
+ }
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#include <osk/mali_osk.h>
+
+#if MALI_DEBUG != 0
+#include <linux/delay.h>
+
+#define TIMER_PERIOD_NS 100
+#define TIMER_TEST_TIME_MS 1000
+typedef struct oskp_time_test
+{
+ osk_timer timer;
+ u32 val;
+ mali_bool should_stop;
+} oskp_time_test;
+
+static mali_bool oskp_timer_has_been_checked = MALI_FALSE;
+#endif
+
+enum hrtimer_restart oskp_timer_callback_wrapper( struct hrtimer * hr_timer )
+{
+ osk_timer *tim;
+
+ tim = CONTAINER_OF( hr_timer, osk_timer, timer );
+ tim->callback( tim->data );
+
+ return HRTIMER_NORESTART;
+}
+
+#if MALI_DEBUG != 0
+static void oskp_check_timer_callback( void *data )
+{
+ oskp_time_test *time_tester = (oskp_time_test*)data;
+
+ (time_tester->val)++;
+
+ if ( time_tester->should_stop == MALI_FALSE )
+ {
+ osk_error err;
+ err = osk_timer_start_ns( &time_tester->timer, TIMER_PERIOD_NS );
+ if ( err != OSK_ERR_NONE )
+ {
+ OSK_PRINT_WARN( OSK_BASE_CORE, "OSK Timer couldn't restart - testing stats will be inaccurate" );
+ }
+ }
+}
+
+void oskp_debug_test_timer_stats( void )
+{
+ oskp_time_test time_tester;
+ osk_ticks start_timestamp;
+ osk_ticks end_timestamp;
+ u32 msec_elapsed;
+ osk_error err;
+
+ if ( oskp_timer_has_been_checked != MALI_FALSE )
+ {
+ return;
+ }
+ oskp_timer_has_been_checked = MALI_TRUE;
+
+ OSK_MEMSET( &time_tester, 0, sizeof(time_tester) );
+
+ err = osk_timer_on_stack_init( &time_tester.timer );
+ if ( err != OSK_ERR_NONE )
+ {
+ goto fail_init;
+ }
+
+ osk_timer_callback_set( &time_tester.timer, &oskp_check_timer_callback, &time_tester );
+
+ start_timestamp = osk_time_now();
+ err = osk_timer_start_ns( &time_tester.timer, TIMER_PERIOD_NS );
+ if ( err != OSK_ERR_NONE )
+ {
+ goto fail_start;
+ }
+
+ msleep( TIMER_TEST_TIME_MS );
+
+ time_tester.should_stop = MALI_TRUE;
+
+ osk_timer_stop( &time_tester.timer );
+ end_timestamp = osk_time_now();
+
+ msec_elapsed = osk_time_elapsed( start_timestamp, end_timestamp );
+
+ OSK_PRINT( OSK_BASE_CORE, "OSK Timer did %d iterations in %dms", time_tester.val, msec_elapsed );
+
+ osk_timer_on_stack_term( &time_tester.timer );
+ return;
+
+ fail_start:
+ osk_timer_on_stack_term( &time_tester.timer );
+ fail_init:
+ OSK_PRINT_WARN( OSK_BASE_CORE, "OSK Timer couldn't init/start for testing stats" );
+ return;
+}
+#endif
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#include <osk/mali_osk.h>
+
+#if MALI_LICENSE_IS_GPL == 0
+/* Wrapper function to allow flushing in Non-GPL driver */
+void oskp_work_func_wrapper( struct work_struct *work )
+{
+ osk_workq_work *osk_work = CONTAINER_OF( work, osk_workq_work, os_work );
+ osk_workq *parent_wq;
+ u32 val;
+ unsigned long flags;
+
+ parent_wq = osk_work->parent_wq;
+ osk_work->actual_fn( osk_work );
+ /* work and osk_work could disappear from this point on */
+
+ /* parent_wq of course shouldn't disappear *yet*, because it must itself flush this function before term */
+
+ spin_lock_irqsave( &parent_wq->active_items_lock, flags );
+ val = --(parent_wq->nr_active_items);
+ /* The operations above and below must form an atomic operation themselves,
+ * hence the lock. See osk_workq_flush() for why */
+ if ( val == 0 )
+ {
+ wake_up( &parent_wq->waitq_zero_active_items );
+ }
+ spin_unlock_irqrestore( &parent_wq->active_items_lock, flags );
+
+ /* parent_wq may've now disappeared */
+}
+
+#endif /* MALI_LICENSE_IS_GPL == 0 */
--- /dev/null
+# Copyright:
+# ----------------------------------------------------------------------------
+# This confidential and proprietary software may be used only as authorized
+# by a licensing agreement from ARM Limited.
+# (C) COPYRIGHT 2010-2012 ARM Limited, ALL RIGHTS RESERVED
+# The entire notice above must be reproduced on all authorized copies and
+# copies may only be made to the extent permitted by a licensing agreement
+# from ARM Limited.
+# ----------------------------------------------------------------------------
+#
+from os.path import basename
+
+Import('env')
+
+# Clone the environment so changes don't affect other build files
+env_osk = env.Clone()
+
+# basenames of files to exclude
+osk_excludes = []
+
+# Do not build mali_osk_timers in commercial build
+if env['mali_license_is_gpl'] == '0':
+ osk_excludes.append('mali_osk_timers.c')
+
+# Source files required for the OSK. We include a "#" in the Glob expression
+# to cause SCons to look in the directory relative that in which the SCons
+# command is executed; otherwise, it start looking for the C source files in
+# the variant directory and will fail to spot changes as the files are not
+# present there.
+osk_src = [Glob('*.c'), Glob('#osk/src/common/*.c')]
+
+# Remove any excluded files
+osk_src = filter(lambda node: basename(node.path) not in osk_excludes, env.Flatten(osk_src))
+
+env_osk.Append( CPPPATH='#osk/src/linux/include' )
+
+if env_osk['v'] != '1':
+ env_osk['MAKECOMSTR'] = '[MAKE] ${SOURCE.dir}'
+
+# Note: cleaning via the Linux kernel build system does not yet work
+if env_osk.GetOption('clean') :
+ makeAction=Action("cd ${SOURCE.dir} && make clean", '$MAKECOMSTR')
+else:
+ makeAction=Action("cd ${SOURCE.dir} && make MALI_DEBUG=${debug} MALI_HW_VERSION=${hwver} MALI_BASE_TRACK_MEMLEAK=${base_qa} MALI_LICENSE_IS_GPL=${mali_license_is_gpl} MALI_USE_UMP=${ump} MALI_UNIT_TEST=${unit} && cp lib.a $STATIC_LIB_PATH/libosk.a", '$MAKECOMSTR')
+
+# The target is libosk.a, built from the source in osk_src, via the action makeAction
+# libosk.a will be copied to $STATIC_LIB_PATH after being built by the standard Linux
+# kernel build system, after which it can be installed to the directory specified if
+# "libs_install" is set; this is done by LibTarget.
+cmd = env_osk.Command('$STATIC_LIB_PATH/libosk.a', osk_src, [makeAction])
+
+# Until we fathom out how the invoke the Linux build system to clean, we can use Clean
+# to remove generated files.
+
+patterns = ['*.o', '*.a', '.*.cmd', 'modules.order', '.tmp_versions', 'Module.symvers']
+
+for p in patterns:
+ Clean(cmd, Glob('#osk/src/linux/%s' % p))
+ Clean(cmd, Glob('#osk/src/common/%s' % p))
+
+env_osk.LibTarget('osk', cmd)
--- /dev/null
+# This confidential and proprietary software may be used only as
+# authorised by a licensing agreement from ARM Limited
+# (C) COPYRIGHT 2010-2011 ARM Limited
+# ALL RIGHTS RESERVED
+# The entire notice above must be reproduced on all authorised
+# copies and copies may only be made to the extent permitted
+# by a licensing agreement from ARM Limited.
+
+Import( 'env' )
+import os
+
+if env['backend'] == 'kernel':
+ SConscript(os.path.join(env['kernel'], 'sconscript'))
+else:
+ SConscript(os.path.join(env['os'] + '_userspace', 'sconscript'))
+ SConscript(os.path.join('userspace', 'sconscript'))
--- /dev/null
+obj-y += src/
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_uk.h
+ * Types and definitions that are common across OSs for both the user
+ * and kernel side of the User-Kernel interface.
+ */
+
+#ifndef _UK_H_
+#define _UK_H_
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif /* __cplusplus */
+
+#include <malisw/mali_stdtypes.h>
+
+/**
+ * @addtogroup base_api
+ * @{
+ */
+
+/**
+ * @defgroup uk_api User-Kernel Interface API
+ *
+ * The User-Kernel Interface abstracts the communication mechanism between the user and kernel-side code of device
+ * drivers developed as part of the Midgard DDK. Currently that includes the Base driver and the UMP driver.
+ *
+ * It exposes an OS independent API to user-side code (UKU) which routes functions calls to an OS-independent
+ * kernel-side API (UKK) via an OS-specific communication mechanism.
+ *
+ * This API is internal to the Midgard DDK and is not exposed to any applications.
+ *
+ * @{
+ */
+
+/**
+ * @brief UK major version
+ */
+#define MALI_MODULE_UK_MAJOR 0
+
+/**
+ * @brief UK minor version
+ */
+#define MALI_MODULE_UK_MINOR 0
+
+/**
+ * These are identifiers for kernel-side drivers implementing a UK interface, aka UKK clients. The
+ * UK module maps this to an OS specific device name, e.g. "gpu_base" -> "GPU0:". Specify this
+ * identifier to select a UKK client to the uku_open() function.
+ *
+ * When a new UKK client driver is created a new identifier needs to be added to the uk_client_id
+ * enumeration and the uku_open() implemenation for the various OS ports need to be updated to
+ * provide a mapping of the identifier to the OS specific device name.
+ *
+ */
+typedef enum uk_client_id
+{
+ /**
+ * Value used to identify the Base driver UK client.
+ */
+ UK_CLIENT_MALI_T600_BASE,
+
+ /** The number of uk clients supported. This must be the last member of the enum */
+ UK_CLIENT_COUNT
+} uk_client_id;
+
+/**
+ * Each function callable through the UK interface has a unique number.
+ * Functions provided by UK clients start from number UK_FUNC_ID.
+ * Numbers below UK_FUNC_ID are used for internal UK functions.
+ */
+typedef enum uk_func
+{
+ UKP_FUNC_ID_CHECK_VERSION, /**< UKK Core internal function */
+ /**
+ * Each UK client numbers the functions they provide starting from
+ * number UK_FUNC_ID. This number is then eventually assigned to the
+ * id field of the uk_header structure when preparing to make a
+ * UK call. See your UK client for a list of their function numbers.
+ */
+ UK_FUNC_ID = 512
+} uk_func;
+
+/**
+ * Arguments for a UK call are stored in a structure. This structure consists
+ * of a fixed size header and a payload. The header carries a 32-bit number
+ * identifying the UK function to be called (see uk_func). When the UKK client
+ * receives this header and executed the requested UK function, it will use
+ * the same header to store the result of the function in the form of a
+ * mali_error return code. The size of this structure is such that the
+ * first member of the payload following the header can be accessed efficiently
+ * on a 32 and 64-bit kernel and the structure has the same size regardless
+ * of a 32 or 64-bit kernel. The uk_kernel_size_type type should be defined
+ * accordingly in the OS specific mali_uk_os.h header file.
+ */
+typedef union uk_header
+{
+ /**
+ * 32-bit number identifying the UK function to be called.
+ * Also see uk_func.
+ */
+ u32 id;
+ /**
+ * The mali_error return code returned by the called UK function.
+ * See the specification of the particular UK function you are
+ * calling for the meaning of the error codes returned. All
+ * UK functions return MALI_ERROR_NONE on success.
+ */
+ mali_error ret;
+ /*
+ * Used to ensure 64-bit alignment of this union. Do not remove.
+ * This field is used for padding and does not need to be initialized.
+ */
+ u64 sizer;
+} uk_header;
+
+/**
+ * This structure carries a 16-bit major and minor number and is sent along with an internal UK call
+ * used during uku_open to identify the versions of the UK module in use by the user-side and kernel-side.
+ */
+typedef struct uku_version_check_args
+{
+ uk_header header; /**< UK call header */
+ u16 major; /**< This field carries the user-side major version on input and the kernel-side major version on output */
+ u16 minor; /**< This field carries the user-side minor version on input and the kernel-side minor version on output. */
+} uku_version_check_args;
+
+/** @} end group uk_api */
+
+/** @} */ /* end group base_api */
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _UK_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_ukk.h
+ * Types and definitions that are common across OSs for the kernel side of the
+ * User-Kernel interface.
+ */
+
+#ifndef _UKK_H_
+#define _UKK_H_
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif /* __cplusplus */
+
+#include <osk/mali_osk.h>
+#include <malisw/mali_stdtypes.h>
+#include <uk/mali_uk.h>
+
+/**
+ * Incomplete definitions of ukk_session, ukk_call_context to satisfy header file dependency in plat/mali_ukk_os.h
+ */
+typedef struct ukk_session ukk_session;
+typedef struct ukk_call_context ukk_call_context;
+#include <plat/mali_ukk_os.h> /* needed for ukkp_session definition */
+
+/**
+ * @addtogroup uk_api User-Kernel Interface API
+ * @{
+ */
+
+/**
+ * @addtogroup uk_api_kernel UKK (Kernel side)
+ * @{
+ *
+ * A kernel-side device driver implements the UK interface with the help of the UKK. The UKK
+ * is an OS independent API for kernel-side code to accept requests from the user-side to
+ * execute functions in the kernel-side device driver.
+ *
+ * A few definitions:
+ * - the kernel-side device driver implementing the UK interface is called the UKK client driver
+ * - the user-side library, application, or driver communicating with the UKK client driver is called the UKU client driver
+ * - the UKK API implementation is called the UKK core. The UKK core is linked with your UKK client driver.
+ *
+ * When a UKK client driver starts it needs to initialize the UKK core by calling ukk_start() and
+ * similarly ukk_stop() when the UKK client driver terminates. A UKK client driver is normally
+ * started by an operating system when a device boots.
+ *
+ * A UKU client driver provides services that are implemented either completely in user-space, kernel-space, or
+ * a combination of both. A UKU client driver makes UK calls to its UKK client driver to execute any functionality
+ * of services that is implemented in kernel-space.
+ *
+ * To make a UK call the UKU client driver needs to establish a connection with the UKK client driver. The UKU API
+ * provides the uku_open() call to establish this connection. Normally this results in the OS calling the open
+ * entry point of the UKK client driver. Here, the UKK client driver needs to initialize a UKK session object
+ * with ukk_session_init() to represent this connection, and register a function that will execute the UK calls
+ * requested over this connection (or UKK session). This function is called the UKK client dispatch handler.
+ *
+ * To prevent the UKU client driver executing an incompatible UK call implementation, the UKK session object
+ * stores the version of the UK calls supported by the function registered to execute the UK calls. As soon as the
+ * UKU client driver established a connection with the UKK client driver, uku_open() makes an internal UK call to
+ * request the version of the UK calls supported by the UKK client driver and will fail if the version expected
+ * by the UKU client driver is not compatible with the version supported by the UKK client driver. Internal UK calls
+ * are handled by the UKK core itself and don't reach the UKK client dispatch handler.
+ *
+ * Once the UKU client driver has established a (compatible) connection with the UKK client driver, the UKU
+ * client driver can execute UK calls by using uku_call(). This normally results in the OS calling the ioctl
+ * handler of your UKK client driver and presenting it with the UK call argument structure that was passed
+ * to uku_call(). It is the responsibility of the ioctl handler to copy the UK call argument structure from
+ * user-space to kernel-space and provide it to the UKK dispatch function, ukk_dispatch(), for execution. Depending
+ * on the particular UK call, the UKK dispatch function will either call the UKK client dispatch handler associated
+ * with the UKK session, or the UKK core dispatch handler if it is an UK internal call. When the UKK dispatch
+ * function returns, the return code of the UK call and the output and input/output parameters in the UK call argument
+ * structure will have been updated. Again, it is the responsibility of the ioctl handler to copy the updated
+ * UK call argument structure from kernel-space back to user-space.
+ *
+ * When the UKK client dispatch handler is called it is passed the UK call argument structure (along with a
+ * UK call context which is discussed later). The UKK client dispatch handler uses the uk_header structure in the
+ * UK call argument structure (which is always the first member in this structure) to determine which UK call in
+ * particular needs to be executed. The uk_header structure is a union of a 32-bit number containing the UK call
+ * function number (as defined by the UKK client driver) and a mali_error return value that will store the return
+ * value of the UK call. The 32-bit UK call function number is normally used to select a particular case in a switch
+ * statement that implements the particular UK call which finally stores the result of the UK call in the mali_error
+ * return value of the uk_header structure.
+ *
+ * A UK call implementation is provided with access to a number of objects it may need during the UK call through
+ * a UKK call context. This UKK call context currently only contains
+ * - a pointer to the UKK session for the UK call
+ *
+ * It is the responsibility of the ioctl handler to initialize a UKK call context using ukk_call_prepare() and pass
+ * it on to the UKK dispatch function. The UK call implementation then uses ukk_session_get() to retrieve the stored
+ * objects in the UKK call context. The UK call implementation normally uses the UKK session pointer returned from
+ * ukk_session_get() to access the UKK client driver's context in which the UKK session is embedded. For example:
+ * struct kbase_context {
+ * int some_kbase_context_data;
+ * int more_kbase_context_data;
+ * ukk_session ukk_session_member;
+ * } *kctx;
+ * kctx = CONTAINER_OF(ukk_session_get(ukk_call_ctx), kbase_context, ukk_session_member);
+ *
+ * A UK call may not use an argument structure with embedded pointers.
+ *
+ * All of this can be translated into the following minimal sample code for a UKK client driver:
+@code
+// Sample code for an imaginary UKK client driver 'TESTDRV' implementing the 'TESTDRV_UK_FOO_FUNC' UK call
+//
+#define TESTDRV_VERSION_MAJOR 0
+#define TESTDRV_VERSION_MINOR 1
+
+typedef enum testdrv_uk_function
+{
+ TESTDRV_UK_FOO_FUNC = (UK_FUNC_ID + 0)
+} testdrv_uk_function;
+
+typedef struct testdrv_uk_foo_args
+{
+ uk_header header;
+ int counters[10]; // input
+ int prev_counters[10]; // output
+} testdrv_uk_foo_args;
+
+typedef struct testdrv_session
+{
+ int counters[10];
+ ukk_session ukk_session_obj;
+} testdrv_session;
+
+testdrv_open(os_driver_context *osctx)
+{
+ testdrv_session *ts;
+ ts = osk_malloc(sizeof(*ts));
+ ukk_session_init(&ts->ukk_session_obj, testdrv_ukk_dispatch, TESTDRV_VERSION_MAJOR, TESTDRV_VERSION_MINOR);
+ osctx->some_field = ts;
+}
+testdrv_close(os_driver_context *osctx)
+{
+ testdrv_session *ts = osctx->some_field;
+ ukk_session_term(&ts->ukk_session_obj)
+ osk_free(ts);
+ osctx->some_field = NULL;
+}
+testdrv_ioctl(os_driver_context *osctx, void *user_arg, u32 args_size)
+{
+ testdrv_session *ts = osctx->some_field;
+ ukk_call_context call_ctx;
+ void *kernel_arg;
+
+ kernel_arg = os_copy_to_kernel_space(user_arg, args_size);
+
+ ukk_call_prepare(&call_ctx, &ts->ukk_session_obj);
+
+ ukk_dispatch(&call_ctx, kernel_arg, args_size);
+
+ os_copy_to_user_space(user_arg, kernel_arg, args_size);
+}
+mali_error testdrv_ukk_dispatch(ukk_call_context *call_ctx, void *arg, u32 args_size)
+{
+ uk_header *header = arg;
+ mali_error ret = MALI_ERROR_FUNCTION_FAILED;
+
+ switch(header->id)
+ {
+ case TESTDRV_UK_FOO_FUNC:
+ {
+ testdrv_uk_foo_args *foo_args = arg;
+ if (sizeof(*foo_args) == args_size)
+ {
+ mali_error result;
+ result = foo(call_ctx, foo_args);
+ header->ret = result;
+ ret = MALI_ERROR_NONE;
+ }
+ break;
+ }
+ }
+ return ret;
+}
+mali_error foo(ukk_call_context *call_ctx, testdrv_uk_foo_args *args) {
+ // foo updates the counters in the testdrv_session object and returns the old counters
+ testdrv_session *session = CONTAINER_OF(ukk_session_get(call_ctx), testdrv_session, ukk_session_obj);
+ memcpy(&args->prev_counters, session->counters, 10 * sizeof(int));
+ memcpy(&session->counters, &args->counters, 10 * sizeof(int));
+ return MALI_ERROR_NONE;
+}
+@endcode
+*/
+
+/**
+ * Maximum size of UK call argument structure supported by UKK clients
+ */
+#define UKK_CALL_MAX_SIZE 512
+
+/**
+ * @brief Dispatch callback of UKK client
+ *
+ * The UKK client's dispatch function is called by UKK core ukk_dispatch()
+ *
+ * The UKK client's dispatch function should return MALI_ERROR_NONE when it
+ * has accepted and executed the UK call. If the UK call is not recognized it
+ * should return MALI_ERROR_FUNCTION_FAILED.
+ *
+ * An example of a piece of code from a UKK client dispatch function:
+ * @code
+ * uk_header *header = (uk_header *)arg;
+ * switch(header->id) {
+ * case MYCLIENT_FUNCTION: {
+ * if (args_size != sizeof(myclient_function_args)) {
+ * return MALI_ERROR_FUNCTION_FAILED; // argument structure size mismatch
+ * } else {
+ * // execute UK call and store result back in header
+ * header->ret = do_my_client_function(ukk_ctx, args);
+ * return MALI_ERROR_NONE;
+ * }
+ * default:
+ * return MALI_ERROR_FUNCTION_FAILED; // UK call function number not recognized
+ * }
+ * @endcode
+ *
+ * For details, see ukk_dispatch().
+ *
+ * Debug builds will assert when a NULL pointer is passed for ukk_ctx or args, or,
+ * args_size < sizeof(uk_header).
+ *
+ * @param[in] ukk_ctx Pointer to a call context
+ * @param[in,out] args Pointer to a argument structure of a UK call
+ * @param[in] args_size Size of the argument structure (in bytes)
+ * @return MALI_ERROR_NONE on success. MALI_ERROR_FUNCTION_FAILED when the UK call was not recognized.
+ */
+typedef mali_error (*ukk_dispatch_function)(ukk_call_context * const ukk_ctx, void * const args, u32 args_size);
+
+/**
+ * Driver session related data for the UKK core.
+ */
+struct ukk_session
+{
+ /**
+ * Session data stored by the OS specific implementation of the UKK core
+ */
+ ukkp_session internal_session;
+
+ /**
+ * UKK client version supported by the call backs provided below - major number of version
+ */
+ u16 version_major;
+
+ /**
+ * UKK client version supported by the call backs provided below - minor number of version
+ */
+ u16 version_minor;
+
+ /**
+ * Function in UKK client that executes UK calls for this UKK session, see
+ * ukk_dispatch_function.
+ */
+ ukk_dispatch_function dispatch;
+};
+
+/**
+ * Stucture containing context data passed in to each UK call. Before each UK call it is initialized
+ * by the ukk_call_prepare() function. UK calls can retrieve the context data using the function
+ * ukk_session_get().
+ */
+struct ukk_call_context
+{
+ /**
+ * Pointer to UKK core session data.
+ */
+ ukk_session *ukk_session;
+};
+
+/**
+ * @brief UKK core startup
+ *
+ * Must be called during the UKK client driver initialization before accessing any UKK provided functionality.
+ *
+ * @return MALI_ERROR_NONE on success. Any other value indicates failure.
+ */
+mali_error ukk_start(void);
+
+/**
+ * @brief UKK core shutdown
+ *
+ * Must be called during the UKK client driver termination to free any resources UKK might have allocated.
+ *
+ * After this has been called no UKK functionality may be accessed.
+ */
+void ukk_stop(void);
+
+/**
+ * @brief Initialize a UKK session
+ *
+ * When a UKK client driver is opened, a UKK session object needs to be initialized to
+ * store information specific to that session with the UKK client driver.
+ *
+ * This UKK session object is normally contained in a session specific data structure created
+ * by the OS specific open entry point of the UKK client driver. The entry point of the
+ * UKK client driver that receives requests from user space to execute UK calls will
+ * need to pass on a pointer to this UKK session object to the ukk_dispatch() function to
+ * execute a UK call for the active session.
+ *
+ * A UKK session supports executing UK calls for a particular version of the UKK client driver
+ * interface. A pointer to the dispatch function that will execute the UK calls needs to
+ * be passed to ukk_session_init(), along with the version (major and minor) of the UKK client
+ * driver interface that this dispatch function supports.
+ *
+ * When the UKK client driver is closed, the initialized UKK session object needs to be
+ * terminated. See ukk_session_term().
+ *
+ * Debug builds will assert when a NULL pointer is passed for ukk_ctx or dispatch.
+ *
+ * @param[out] ukk_session Pointer to UKK session to initialize
+ * @param[in] dispatch Pointer to dispatch function to associate with the UKK session
+ * @param[in] version_major Dispatch function will handle UK calls for this major version
+ * @param[in] version_minor Dispatch function will handle UK calls for this minor version
+ * @return MALI_ERROR_NONE on success. Any other value indicates failure.
+ */
+mali_error ukk_session_init(ukk_session *ukk_session, ukk_dispatch_function dispatch, u16 version_major, u16 version_minor);
+
+/**
+ * @brief Terminates a UKK session
+ *
+ * Frees any resources allocated for the UKK session object. No UK calls for this session
+ * may be executing when calling this function. This function invalidates the UKK session
+ * object and must not be used anymore until it is initialized again with ukk_session_init().
+ *
+ * Debug builds will assert when a NULL pointer is passed for ukk_session.
+ *
+ * @param[in,out] ukk_session Pointer to UKK session to terminate
+ */
+void ukk_session_term(ukk_session *ukk_session);
+
+/**
+ * @brief Prepare a context in which to execute a UK call
+ *
+ * UK calls are passed a call context that allows them to get access to the UKK session data.
+ * Given a call context, UK calls use ukk_session_get() to get access to the UKK session data.
+ *
+ * Debug builds will assert when a NULL pointer is passed for ukk_ctx, ukk_session.
+ *
+ * @param[out] ukk_ctx Pointer to call context to initialize.
+ * @param[in] ukk_session Pointer to UKK session to associate with the call context
+ */
+void ukk_call_prepare(ukk_call_context * const ukk_ctx, ukk_session * const ukk_session);
+
+/**
+ * @brief Get the UKK session of a call context
+ *
+ * Returns the UKK session associated with a call context. See ukk_call_prepare.
+ *
+ * Debug builds will assert when a NULL pointer is passed for ukk_ctx.
+ *
+ * @param[in] ukk_ctx Pointer to call context
+ * @return Pointer to UKK session associated with the call context
+ */
+void *ukk_session_get(ukk_call_context * const ukk_ctx);
+
+/**
+ * @brief Copy data from user space to kernel space
+ *
+ * @param[in] bytes Number of bytes to copy from @ref user_buffer to @ref_kernel_buffer
+ * @param[out] kernel_buffer Pointer to data buffer in kernel space.
+ * @param[in] user_buffer Pointer to data buffer in user space.
+ *
+ * @return Returns MALI_ERROR_NONE on success.
+ */
+
+mali_error ukk_copy_from_user( size_t bytes, void * kernel_buffer, const void * const user_buffer );
+
+/**
+ * @brief Copy data from kernel space to user space
+ *
+ * @param[in] bytes Pointer to a call context
+ * @param[in] user_buffer Pointer to a call context
+ * @param[out] kernel_buffer Pointer to a call context
+ *
+ * @return Returns MALI_ERROR_NONE on success.
+ */
+
+mali_error ukk_copy_to_user( size_t bytes, void * user_buffer, const void * const kernel_buffer );
+
+/**
+ * @brief Dispatch a UK call
+ *
+ * Dispatches the UK call to the UKK client or the UKK core in case of an internal UK call. The id field
+ * in the header field of the argument structure identifies which UK call needs to be executed. Any
+ * UK call with id equal or larger than UK_FUNC_ID is dispatched to the UKK client.
+ *
+ * If the UK call was accepted by the dispatch handler of the UKK client or UKK core, this function returns
+ * with MALI_ERROR_NONE and the result of executing the UK call is stored in the header.ret field of the
+ * in the argument structure. This function returns MALI_ERROR_FUNCTION_FAILED when the UK call is not
+ * accepted by the dispatch handler.
+ *
+ * If a UK call fails while executing in the dispatch handler of the UKK client or UKK core
+ * the UK call is reponsible for cleaning up any resources it allocated up to the point a failure
+ * occurred.
+ *
+ * Before accepting a UK call, the dispatch handler of the UKK client or UKK core should compare the
+ * the size of the argument structure based on the function id in header.id with the args_size parameter.
+ * Only if they match the UK call should be attempted, otherwise MALI_ERROR_FUNCTION_FAILED
+ * should be returned.
+ *
+ * An example of a piece of code from a UKK client dispatch handler:
+ * @code
+ * uk_header *header = (uk_header *)arg;
+ * switch(header->id) {
+ * case MYCLIENT_FUNCTION: {
+ * if (args_size != sizeof(myclient_function_args)) {
+ * return MALI_ERROR_FUNCTION_FAILED; // argument structure size mismatch
+ * } else {
+ * // execute UK call and store result back in header
+ * header->ret = do_my_client_function(ukk_ctx, args);
+ * return MALI_ERROR_NONE;
+ * }
+ * default:
+ * return MALI_ERROR_FUNCTION_FAILED; // UK call function number not recognized
+ * }
+ * @endcode
+ *
+ * Debug builds will assert when a NULL pointer is passed for ukk_ctx, args or args_size
+ * is < sizeof(uk_header).
+ *
+ * @param[in] ukk_ctx Pointer to a call context
+ * @param[in,out] args Pointer to a argument structure of a UK call
+ * @param[in] args_size Size of the argument structure (in bytes)
+ * @return MALI_ERROR_NONE on success. MALI_ERROR_FUNCTION_FAILED when the UK call was not accepted
+ * by the dispatch handler of the UKK client or UKK core, or the passed in argument structure
+ * is not large enough to store the required uk_header structure.
+ */
+mali_error ukk_dispatch(ukk_call_context * const ukk_ctx, void * const args, u32 args_size);
+
+/** @} end group uk_api_kernel */
+
+/** @} end group uk_api */
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _UKK_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_uku.h
+ * Types and definitions that are common across OSs for the user side of the
+ * User-Kernel interface.
+ */
+
+#ifndef _UKU_H_
+#define _UKU_H_
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif /* __cplusplus */
+
+#include <malisw/mali_stdtypes.h>
+#include <uk/mali_uk.h>
+#include <plat/mali_uk_os.h>
+
+/**
+ * @addtogroup uk_api User-Kernel Interface API
+ * @{
+ */
+
+/**
+ * @defgroup uk_api_user UKU (User side)
+ *
+ * The UKU is an OS independent API for user-side code which provides functions to
+ * - open and close a UKK client driver, a kernel-side device driver implementing the UK interface
+ * - call functions inside a UKK client driver
+ *
+ * The code snippets below show an example using the UKU API:
+ *
+ * Start with opening the imaginary Midgard Base UKK client driver
+ *
+@code
+ mali_error ret;
+ uku_context uku_ctx;
+ uku_client_version client_version;
+ uku_open_status open_status;
+
+ // open a user-kernel context
+ client_version.major = TESTDRV_UK_MAJOR;
+ client_version.minor = TESTDRV_UK_MINOR;
+ open_status = uku_open(UK_CLIENT_MALI_T600_BASE, &client_version, &uku_ctx);
+ if (UKU_OPEN_OK != open_status)
+ {
+ mali_tpi_printf("failed to open a user-kernel context\n");
+ goto cleanup;
+ }
+@endcode
+ *
+ * We are going to call a function foo in the Midgard Base UKK client driver. For sample purposes this
+ * function foo will simply double the provided input value.
+ *
+ * First we setup the header of the argument structure to identify that we are calling the
+ * a function foo in the Midgard Base UKK client driver identified with the id BASE_UK_FOO_FUNC.
+ *
+@code
+ base_uk_foo_args foo_args;
+ foo_args.header.id = BASE_UK_FOO_FUNC;
+@endcode
+ *
+ * Followed by the setup of the arguments for the function foo.
+ *
+@code
+ foo_args.input_value = 48;
+@endcode
+ *
+ * Then we use UKU to actually call the function in the Midgard Base UKK client driver.
+ *
+@code
+ // call kernel-side foo function
+ ret = uku_call(uku_ctx, &foo_args, sizeof(foo_args));
+ if (MALI_ERROR_NONE == ret && MALI_ERROR_NONE == foo_args.header.ret)
+ {
+@endcode
+ *
+ * If the uku_call() function succeeded we can check the return code of the foo function. The return
+ * value is returned in the ret field of the header. If it succeeded, we verify here that the
+ * foo function indeed doubled the input value.
+ *
+@code
+ // retrieve data returned by kernel-side foo function
+ mali_tpi_printf("foo returned value %d\n", foo_args.output_value);
+
+ // output_value should match input_value * 2
+ if (foo_args.output_value != foo_args.input_value * 2)
+ {
+ // data didn't match: test fails
+ ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ }
+@endcode
+ *
+ * When we are done, we close the Midgard Base UKK client.
+ *
+@code
+cleanup:
+ // close an opened user-kernel context
+ if (UKU_OPEN_OK == open_status)
+ {
+ uku_close(uku_ctx);
+ }
+@endcode
+ * @{
+ */
+
+/**
+ * User-side representation of a connection with a UKK client.
+ * See uku_open for opening a connection, and uku_close for closing
+ * a connection.
+ */
+/* Some compilers require a forward declaration of the structure */
+struct uku_context;
+typedef struct uku_context uku_context;
+
+/**
+ * Status returned from uku_open() as a result of trying to open a connection to a UKK client
+ */
+typedef enum uku_open_status
+{
+ UKU_OPEN_OK, /**< UKK client opened succesfully and versions are compatible */
+ UKU_OPEN_INCOMPATIBLE, /**< UKK client opened succesfully but versions are not compatible and the UKK client
+ connection was closed. */
+ UKU_OPEN_FAILED /**< Could not open UKK client or UKK client failed to perform version check */
+} uku_open_status;
+
+/**
+ * This structure carries a 16-bit major and minor number and is provided to a uku_open call
+ * to identify the versions of the UK client of the user-side on input, and on output the
+ * version of the UK client on the kernel-side. See uku_open.
+ */
+typedef struct uku_client_version
+{
+ /**
+ * 16-bit number identifying the major version. Interfaces with different major version numbers
+ * are incompatible. This field carries the user-side major version on input and the kernel-side
+ * major version on output.
+ */
+ u16 major;
+ /**
+ * 16-bit number identifying the minor version. A user-side interface minor version that is equal
+ * to or less than the kernel-side interface minor version is compatible. A user-side interface
+ * minor version that is greater than the kernel-side interface minor version is incompatible
+ * (as it is requesting more functionality than exists). This field carries the user-side minor
+ * version on input and the kernel-side minor version on output.
+ */
+ u16 minor;
+} uku_client_version;
+
+/**
+ * @brief Open a connection to a UKK client
+ *
+ * The User-Kernel interface communicates with a kernel-side driver over
+ * an OS specific communication channel. A UKU context object stores the
+ * necessary OS specific objects and state information to represent this
+ * OS specific communication channel. A UKU context, defined by the
+ * uku_context type is passed in as the first argument to nearly all
+ * UKU functions. These UKU functions expect the UKU context to be valid,
+ * an invalid UKU context will trigger a debug assert (in debug builds).
+ *
+ * The function uku_open() opens a connection to a kernel-side driver with
+ * a User-Kernel interface, aka UKK client, and returns an initialized
+ * UKU context. The function uku_close() closes the connection to the UKK
+ * client.
+ *
+ * The kernel-side driver may support multiple instances and the particular
+ * instance that needs to be opened is selected by the instance argument.
+ * An instance of a kernel-side driver is normally associated with a particular
+ * instance of a physical hardware block, e.g. each instance corresponds
+ * to one of the ports of a UART controller. See the specifics of the
+ * kernel-side driver to find out which instances are supported.
+ *
+ * As the user and kernel-side of the UK interface are physically two
+ * different entities, they might end up using different versions of the
+ * UK interface and therefore part of the opening process makes an
+ * internal UK call to verify if the versions are compatible. A version
+ * has a major and minor part. Interfaces with different major version
+ * numbers are incompatible. A user-side interface minor version that is equal
+ * to or less than the kernel-side interface minor version is compatible.
+ * A user-side interface minor version that is greater than the kernel-side
+ * interface minor version is incompatible (as it is requesting more
+ * functionality than exists).
+ *
+ * Each UKK client has a unique id as defined by the uk_client_id
+ * enumeration. These IDs are mapped to OS specific device file names
+ * that refer to their respective kernel device drivers. Any new UKK client
+ * needs to be added to the uk_client_id enumeration.
+ *
+ * A UKU context must be shareable between threads. It may be shareable
+ * between processes. This attribute is mostly dependent on the OS specific
+ * communication channel used to communicate with the UKK client. When
+ * multiple threads use the same UKU context only one should be responsible
+ * for closing it and ensuring the other threads are not using it anymore.
+ *
+ * Opening a UKU context is considered to be an expensive operation, most
+ * likely resulting in loading a kernel device driver when opened for
+ * the first time in a system. The specific kernel device driver to be
+ * opened is defined by the OS specific implementation of this function and
+ * is not configurable.
+ *
+ * Once a UKU context is opened and in use by the user-kernel interface it
+ * is expected to operate without error. Any communication error will not
+ * result in an attempt by the user-kernel interface implementation to
+ * re-establish the OS specific communication channel and is considered
+ * to be a non-recoverable fault.
+ *
+ * Notes on Base driver context and UKU context
+ *
+ * A UKU context is opened each time a base driver context is created.
+ * A UKU context and base driver context therefore have a 1:1 relationship.
+ * A base driver context represents an isolated GPU address space and because
+ * of the 1:1 relationship with a UKU context, a UKU context can also be seen
+ * to present an isolated GPU address space. Each process is currently
+ * expected to create one base driver context (and therefore open one UKU
+ * context per process), but this might change, having multiple base driver
+ * contexts open per process, in case we need to support WebGL, where each
+ * GLES context must use a separate GPU address space inside the web browser
+ * to prevent seperate brower tabs from interfering with each other.
+ *
+ * Debug builds will assert when a NULL pointer is passed for the version or
+ * uku_ctx parameters, or when an unknown enumerated value for the id parameter
+ * is used.
+ *
+ * @param[in] id UKK client identifier, see uk_client_id.
+ * @param[in] instance instance number (0..) of the UKK client driver
+ * @param[in,out] version of the UKU client on input. On output it contains the version
+ * of the UKK client, when uku_open returns UKU_OPEN_OK or UKU_OPEN_INCOMPATIBLE,
+ * otherwise the output value is undefined.
+ * @param[out] uku_ctx Pointer to User-Kernel context to initialize
+ * @return UKU_OPEN_OK when the connection to the UKK client was successful.
+ * @return UKU_OPEN_FAILED when the connection to the UKK client could not be established,
+ * or the UKK client failed to perform version verification.
+ * @return UKU_OPEN_INCOMPATIBLE when the version of the UKK and UKU clients are incompatible.
+ */
+uku_open_status uku_open(uk_client_id id, u32 instance, uku_client_version *version, uku_context *uku_ctx) CHECK_RESULT;
+
+/**
+ * @brief Returns OS specific driver context from UKU context
+ *
+ * The UKU context abstracts the connection to a kernel-side driver. If OS specific code
+ * needs to communicate with this kernel-side driver directly, it can use this function
+ * to retrieve the OS specific object hidden by the UKU context. This object must only
+ * be used while the UKU context is open and must only be used by OS specific code.
+ *
+ * Debug builds will assert when a NULL pointer is passed for uku_ctx.
+ *
+ * @param[in] uku_ctx Pointer to a valid User-Kernel context. See uku_open.
+ * @return OS specific driver context, e.g. for Linux this would be an integer
+ * representing a file descriptor.
+ *
+ */
+void *uku_driver_context(uku_context *uku_ctx) CHECK_RESULT;
+
+/**
+ * @brief Closes a connection to a UKK client
+ *
+ * Closes a previously opened connection to a kernel-side driver with a
+ * User-Kernel interface, aka UKK client. The UKU context uku_ctx
+ * has now become invalid.
+ *
+ * Before calling this function, any UKU function using the UKU context
+ * uku_ctx must have finished. The UKU context uku_ctx must not be in
+ * use.
+ *
+ * Debug builds will assert when a NULL pointer is passed for uku_ctx.
+ *
+ * @param[in] uku_ctx Pointer to a valid User-Kernel context. See uku_open.
+ */
+void uku_close(uku_context * const uku_ctx);
+
+/**
+ * @brief Calls a function in a UKK client
+ *
+ * A UKK client defines a structure for each function callable over the UK
+ * interface. The structure starts with a header field of type uk_header,
+ * followed by the arguments for the function, e.g.
+ *
+ * @code
+ * typedef struct base_uk_foo_args
+ * {
+ * uk_header header; // first member is the header
+ * int n; // followed by function arguments
+ * int doubled_n;
+ * ...
+ * } base_uk_foo_args;
+ * @endcode
+ *
+ * The header.id field identifies the function to be called. See the UKK
+ * client documentation for a list of available functions and the structure
+ * definitions associated with them.
+ *
+ * The arguments in the structure can be of type input, input/output or
+ * output. All input and input/output arguments must be initialized in the
+ * structure before calling the uku_call() function. Memory pointed to by
+ * pointers in the structure should at least remain allocated until uku_call
+ * returns.
+ *
+ * When uku_call has successfully executed the function in the UKK client,
+ * it stores the return code of the function in the header.ret field, and
+ * only in this case the output and input/output members are considered to
+ * be valid in the structure.
+ *
+ * For example, to call function 'foo' which simply doubles the supplied
+ * argument 'n':
+ * @code
+ * base_uk_foo_args args;
+ * mali_error ret;
+ *
+ * args.header.id = BASE_UK_FOO_FUNC;
+ * args.n = 10;
+ *
+ * ret = uku_call(uku_ctx, &args, sizeof(args));
+ * if (MALI_ERROR_NONE == ret)
+ * {
+ * if (MALI_ERROR_NONE == args.header.ret)
+ * {
+ * printf("%d*%d=%d\n", args.n, args.n, args.doubled_n)
+ * }
+ * }
+ * @endcode
+ *
+ * Debug builds will assert when a NULL pointer is passed for uku_ctx or
+ * args, or args_size < sizeof(uku_header).
+ *
+ * @param[in] uku_ctx Pointer to a valid User-Kernel context. See uku_open.
+ * @param[in,out] args Pointer to an argument structure associated with
+ * the function to be called in the UKK client.
+ * @param[in] args_size Size of the argument structure in bytes
+ * @return MALI_ERROR_NONE on success. Any other value indicates failure,
+ * and the structure pointed to by args may contain invalid information.
+ */
+mali_error uku_call(uku_context *uku_ctx, void *args, u32 args_size) CHECK_RESULT;
+
+
+/** @} end group uk_api_user */
+
+/** @} end group uk_api */
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _UKU_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <fcntl.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/ioctl.h>
+#include <pthread.h>
+#include <errno.h>
+#include <linux/version.h>
+
+#include <uk/mali_uku.h>
+#include <cdbg/mali_cdbg.h>
+#include <cutils/cstr/mali_cutils_cstr.h>
+#include <cutils/linked_list/mali_cutils_slist.h>
+#include <stdlib/mali_stdlib.h>
+
+#define LINUX_MALI_DEVICE_NAME "/dev/mali"
+
+#if CSTD_OS_ANDROID == 0 /* Android 2.3.x doesn't support pthread_atfork */
+
+/** Datastructures to keep track of all open file descriptors to UK clients for this process */
+struct _fd_admin
+{
+ pthread_mutex_t fd_admin_mutex; /** protects access to this datastructure */
+ mali_bool atfork_registered; /** MALI_TRUE when an atfork handler has been installed for this process */
+ cutils_slist fd_list; /** list tracking all open file descriptors to UK clients for this process */
+};
+STATIC struct _fd_admin fd_admin =
+{
+ PTHREAD_MUTEX_INITIALIZER,
+ MALI_FALSE,
+ {{NULL,NULL}}
+};
+typedef struct fd_list_item
+{
+ cutils_slist_item link;
+ int fd;
+} fd_list_item;
+
+
+/** atfork handler called in child's context to close all open file descriptors to UK clients in the child */
+STATIC void ukup_fd_child_atfork_handler(void)
+{
+ fd_list_item *item;
+
+ /* close all file descriptors registered with ukup_add_file_descriptor() */
+ CUTILS_SLIST_FOREACH(&fd_admin.fd_list, fd_list_item, link, item)
+ {
+ close(item->fd);
+ }
+}
+
+/** removes and closes a file descriptor added to the list of open file descriptors to UK clients
+ * by ukup_fd_add() earlier
+ */
+STATIC void ukup_fd_remove_and_close(int fd)
+{
+ int rc;
+ fd_list_item *item = NULL;
+
+ rc = pthread_mutex_lock(&fd_admin.fd_admin_mutex);
+ if (rc != 0)
+ {
+ CDBG_PRINT_INFO(CDBG_BASE, "can't lock file descriptor list, error %d\n", errno);
+ goto exit_mutex_lock;
+ }
+
+ CUTILS_SLIST_FOREACH(&fd_admin.fd_list, fd_list_item, link, item)
+ {
+ if (item->fd == fd)
+ {
+ CUTILS_SLIST_REMOVE(&fd_admin.fd_list, item, link);
+ stdlib_free(item);
+ close(fd);
+ break;
+ }
+ }
+
+ pthread_mutex_unlock(&fd_admin.fd_admin_mutex);
+
+ CDBG_ASSERT_MSG(CUTILS_SLIST_IS_VALID(item, link), "file descriptor %d not found on list!\n", fd);
+
+exit_mutex_lock:
+ return;
+}
+
+
+/** add a file descriptor to the list of open file descriptors to UK clients */
+STATIC mali_bool ukup_fd_add(int fd)
+{
+ int rc;
+ fd_list_item *item = NULL;
+
+ rc = pthread_mutex_lock(&fd_admin.fd_admin_mutex);
+ if (rc != 0)
+ {
+ CDBG_PRINT_INFO(CDBG_BASE, "can't lock file descriptor list, error %d\n", errno);
+ goto exit_mutex_lock;
+ }
+
+ if (MALI_FALSE == fd_admin.atfork_registered)
+ {
+ CUTILS_SLIST_INIT(&fd_admin.fd_list);
+
+ rc = pthread_atfork(NULL, NULL, ukup_fd_child_atfork_handler);
+ if (rc != 0)
+ {
+ CDBG_PRINT_INFO(CDBG_BASE, "pthread_atfork failed, error %d\n", errno);
+ goto exit;
+ }
+
+ fd_admin.atfork_registered = MALI_TRUE;
+ }
+
+ item = stdlib_malloc(sizeof(fd_list_item));
+ if (NULL == item)
+ {
+ CDBG_PRINT_INFO(CDBG_BASE, "allocating file descriptor list item failed\n");
+ goto exit;
+ }
+
+ item->fd = fd;
+
+ CUTILS_SLIST_PUSH_FRONT(&fd_admin.fd_list, item, fd_list_item, link);
+
+ pthread_mutex_unlock(&fd_admin.fd_admin_mutex);
+
+ return MALI_TRUE;
+
+exit:
+ if (NULL != item)
+ {
+ stdlib_free(item);
+ }
+
+ pthread_mutex_unlock(&fd_admin.fd_admin_mutex);
+exit_mutex_lock:
+ return MALI_FALSE;
+}
+
+#endif /* CSTD_OS_ANDROID == 0 */
+
+
+uku_open_status uku_open(uk_client_id id, u32 instance, uku_client_version *version, uku_context *uku_ctx)
+{
+ const char *linux_device_name;
+ char format_device_name[16];
+ struct stat filestat;
+ int fd;
+ uku_version_check_args version_check_args;
+ mali_error err;
+
+ CDBG_ASSERT_POINTER(version);
+ CDBG_ASSERT_POINTER(uku_ctx);
+
+ if(CDBG_SIMULATE_FAILURE(CDBG_BASE))
+ {
+ return UKU_OPEN_FAILED;
+ }
+
+ switch(id)
+ {
+ case UK_CLIENT_MALI_T600_BASE:
+ cutils_cstr_snprintf(format_device_name, sizeof(format_device_name), "%s%d", LINUX_MALI_DEVICE_NAME, instance);
+ linux_device_name = format_device_name;
+ break;
+ default:
+ CDBG_ASSERT_MSG(MALI_FALSE, "invalid uk_client_id value (%d)\n", id);
+ return UKU_OPEN_FAILED;
+ }
+
+ /* open the kernel device driver */
+ fd = open(linux_device_name, O_RDWR|O_CLOEXEC);
+
+ if (-1 == fd)
+ {
+ CDBG_PRINT_INFO(CDBG_BASE, "failed to open device file %s\n", linux_device_name);
+ return UKU_OPEN_FAILED;
+ }
+
+ /* query the file for information */
+ if (0 != fstat(fd, &filestat))
+ {
+ close(fd);
+ CDBG_PRINT_INFO(CDBG_BASE, "failed to query device file %s for type information\n", linux_device_name);
+ return UKU_OPEN_FAILED;
+ }
+
+ /* verify that it is a character special file */
+ if (0 == S_ISCHR(filestat.st_mode))
+ {
+ close(fd);
+ CDBG_PRINT_INFO(CDBG_BASE, "file %s is not a character device file", linux_device_name);
+ return UKU_OPEN_FAILED;
+ }
+
+ /* use the internal UK call UKP_FUNC_ID_CHECK_VERSION to verify versions */
+ version_check_args.header.id = UKP_FUNC_ID_CHECK_VERSION;
+ version_check_args.major = version->major;
+ version_check_args.minor = version->minor;
+
+ uku_ctx->ukup_internal_struct.fd = fd;
+ err = uku_call(uku_ctx, &version_check_args, sizeof(version_check_args));
+ if (MALI_ERROR_NONE == err && MALI_ERROR_NONE == version_check_args.header.ret)
+ {
+ mali_bool incompatible =
+ ( (version->major != version_check_args.major) || (version->minor > version_check_args.minor) );
+
+ if (incompatible)
+ {
+ CDBG_PRINT_INFO(CDBG_BASE, "file %s is not of a compatible version (user %d.%d, kernel %d.%d)\n",
+ linux_device_name, version->major, version->minor, version_check_args.major, version_check_args.minor);
+ }
+
+ /* output kernel-side version */
+ version->major = version_check_args.major;
+ version->minor = version_check_args.minor;
+
+ if (incompatible)
+ {
+ uku_ctx->ukup_internal_struct.fd = -1;
+ close(fd);
+ return UKU_OPEN_INCOMPATIBLE;
+ }
+ }
+ else
+ {
+ close(fd);
+ return UKU_OPEN_FAILED;
+ }
+
+#ifdef MALI_DEBUG
+ uku_ctx->ukup_internal_struct.canary = MALI_UK_CANARY_VALUE;
+#endif
+
+#if CSTD_OS_ANDROID == 0
+ /* track all open file descriptors to UK clients */
+ if (!ukup_fd_add(fd))
+ {
+ close(fd);
+ return UKU_OPEN_FAILED;
+ }
+#endif
+
+ return UKU_OPEN_OK;
+}
+
+
+void *uku_driver_context(uku_context *uku_ctx)
+{
+ CDBG_ASSERT_POINTER(uku_ctx);
+ return (void *)&uku_ctx->ukup_internal_struct.fd;
+}
+
+void uku_close(uku_context *uku_ctx)
+{
+ CDBG_ASSERT_POINTER(uku_ctx);
+#ifdef MALI_DEBUG
+ CDBG_ASSERT(uku_ctx->ukup_internal_struct.canary == MALI_UK_CANARY_VALUE);
+ uku_ctx->ukup_internal_struct.canary = 0;
+#endif
+#if CSTD_OS_ANDROID == 0
+ ukup_fd_remove_and_close(uku_ctx->ukup_internal_struct.fd);
+#else
+ close(uku_ctx->ukup_internal_struct.fd);
+#endif
+}
+
+mali_error uku_call(uku_context *uku_ctx, void *args, u32 args_size)
+{
+ uk_header *header = (uk_header *)args;
+ u32 cmd;
+
+ CDBG_ASSERT_POINTER(uku_ctx);
+ CDBG_ASSERT_POINTER(args);
+ CDBG_ASSERT_MSG(args_size >= sizeof(uk_header), "argument structure not large enough to contain required uk_header\n");
+
+ if(CDBG_SIMULATE_FAILURE(CDBG_BASE))
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+
+ cmd = _IOC(_IOC_READ|_IOC_WRITE, LINUX_UK_BASE_MAGIC, header->id, args_size);
+
+ /* call ioctl handler of driver */
+ if (0 != ioctl(uku_ctx->ukup_internal_struct.fd, cmd, args))
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+ else
+ {
+ return MALI_ERROR_NONE;
+ }
+}
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2011 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_uk_os.h
+ * User-Kernel Interface (kernel and user-side) dependent APIs (Linux).
+ */
+
+#ifndef _UK_OS_H_ /* Linux */
+#define _UK_OS_H_
+
+#ifdef MALI_DEBUG
+#define MALI_UK_CANARY_VALUE 0xb2bdbdf6
+#endif
+
+#if MALI_BACKEND_KERNEL
+
+#define LINUX_UK_BASE_MAGIC 0x80 /* BASE UK ioctl */
+
+struct uku_context
+{
+ struct
+ {
+#ifdef MALI_DEBUG
+ u32 canary;
+#endif
+ int fd;
+ } ukup_internal_struct;
+};
+
+#else /* MALI_BACKEND_KERNEL */
+
+typedef struct ukk_userspace
+{
+ void * ctx;
+ mali_error (*dispatch)(void * /*ctx*/, void* /*msg*/, u32 /*size*/);
+ void (*close)(struct ukk_userspace * /*self*/);
+} ukk_userspace;
+
+typedef ukk_userspace * (*kctx_open)(void);
+
+struct uku_context
+{
+ struct
+ {
+#ifdef MALI_DEBUG
+ u32 canary;
+#endif
+ ukk_userspace * ukku;
+ } ukup_internal_struct;
+};
+
+#endif /* MALI_BACKEND_KERNEL */
+
+#endif /* _UK_OS_H_ */
--- /dev/null
+/*
+ * This confidential and proprietary soft:q!ware may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+/**
+ * @file mali_ukk_os.h
+ * Types and definitions that are common for Linux OSs for the kernel side of the
+ * User-Kernel interface.
+ */
+
+#ifndef _UKK_OS_H_ /* Linux version */
+#define _UKK_OS_H_
+
+#include <linux/fs.h>
+
+/**
+ * @addtogroup uk_api User-Kernel Interface API
+ * @{
+ */
+
+/**
+ * @addtogroup uk_api_kernel UKK (Kernel side)
+ * @{
+ */
+
+/**
+ * Internal OS specific data structure associated with each UKK session. Part
+ * of a ukk_session object.
+ */
+typedef struct ukkp_session
+{
+ int dummy; /**< No internal OS specific data at this time */
+} ukkp_session;
+
+/** @} end group uk_api_kernel */
+
+/** @} end group uk_api */
+
+#endif /* _UKK_OS_H__ */
--- /dev/null
+# This confidential and proprietary software may be used only as
+# authorised by a licensing agreement from ARM Limited
+# (C) COPYRIGHT 2010 ARM Limited
+# ALL RIGHTS RESERVED
+# The entire notice above must be reproduced on all authorised
+# copies and copies may only be made to the extent permitted
+# by a licensing agreement from ARM Limited.
+
+Import( 'env' )
+
+libs=env.StaticLibrary( '$STATIC_LIB_PATH/uku', ['mali_uku_linux.c'] )
+
+env.LibTarget('uku', libs)
+
+if env.has_key('libs_install'):
+ env.Install (env['libs_install'], libs)
+ env.Alias ('libs', env['libs_install'])
+else:
+ env.Alias ('libs', libs)
--- /dev/null
+obj-y += ukk/
--- /dev/null
+# Copyright:
+# ----------------------------------------------------------------------------
+# This confidential and proprietary software may be used only as authorized
+# by a licensing agreement from ARM Limited.
+# (C) COPYRIGHT 2010 ARM Limited, ALL RIGHTS RESERVED
+# The entire notice above must be reproduced on all authorized copies and
+# copies may only be made to the extent permitted by a licensing agreement
+# from ARM Limited.
+# ----------------------------------------------------------------------------
+#
+
+SConscript('ukk/sconscript')
--- /dev/null
+ccflags-$(CONFIG_VITHAR) += -DMALI_DEBUG=1 -DMALI_HW_TYPE=2 \
+ -DMALI_USE_UMP=0 -DMALI_HW_VERSION=r0p0 -DMALI_BASE_TRACK_MEMLEAK=0 \
+ -DMALI_ANDROID=1 -DMALI_ERROR_INJECT_ON=0 -DMALI_NO_MALI=0 -DMALI_BACKEND_KERNEL=1 \
+ -DMALI_FAKE_PLATFORM_DEVICE=1 -DMALI_MOCK_TEST=0 -DMALI_KERNEL_TEST_API=0 \
+ -DMALI_INFINITE_CACHE=0 -DMALI_LICENSE_IS_GPL=1 -DMALI_PLATFORM_CONFIG=exynos5 \
+ -DMALI_UNIT_TEST=0 -DMALI_GATOR_SUPPORT=0 -DUMP_LICENSE_IS_GPL=1 \
+ -DUMP_SVN_REV_STRING="\"dummy\"" -DMALI_RELEASE_NAME="\"dummy\""
+
+ROOTDIR = $(src)/../../..
+
+ccflags-$(CONFIG_VITHAR) += -I$(ROOTDIR)/kbase
+ccflags-$(CONFIG_VITHAR) += -I$(ROOTDIR)
+
+ccflags-y += -I$(ROOTDIR) -I$(ROOTDIR)/include -I$(ROOTDIR)/osk/src/linux/include -I$(ROOTDIR)/uk/platform_dummy
+ccflags-y += -I$(ROOTDIR)/kbase/midg_gpus/r0p0
+
+obj-y += mali_ukk.o
+
+obj-y += linux/
--- /dev/null
+ccflags-$(CONFIG_VITHAR) += -DMALI_DEBUG=1 -DMALI_HW_TYPE=2 \
+ -DMALI_USE_UMP=0 -DMALI_HW_VERSION=r0p0 -DMALI_BASE_TRACK_MEMLEAK=0 \
+ -DMALI_ANDROID=1 -DMALI_ERROR_INJECT_ON=0 -DMALI_NO_MALI=0 -DMALI_BACKEND_KERNEL=1 \
+ -DMALI_FAKE_PLATFORM_DEVICE=1 -DMALI_MOCK_TEST=0 -DMALI_KERNEL_TEST_API=0 \
+ -DMALI_INFINITE_CACHE=0 -DMALI_LICENSE_IS_GPL=1 -DMALI_PLATFORM_CONFIG=exynos5 \
+ -DMALI_UNIT_TEST=0 -DMALI_GATOR_SUPPORT=0 -DUMP_LICENSE_IS_GPL=1 \
+ -DUMP_SVN_REV_STRING="\"dummy\"" -DMALI_RELEASE_NAME="\"dummy\""
+
+ROOTDIR = $(src)/../../../..
+
+ccflags-$(CONFIG_VITHAR) += -I$(ROOTDIR)/kbase
+ccflags-$(CONFIG_VITHAR) += -I$(ROOTDIR)
+
+ccflags-y += -I$(ROOTDIR) -I$(ROOTDIR)/include -I$(ROOTDIR)/osk/src/linux/include -I$(ROOTDIR)/uk/platform_dummy
+ccflags-y += -I$(ROOTDIR)/kbase/midg_gpus/r0p0
+
+obj-y += mali_ukk_os.o
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#include <linux/module.h> /* Needed by all modules */
+#include <linux/kernel.h> /* Needed for KERN_INFO */
+#include <linux/init.h> /* Needed for the macros */
+
+#include <osk/mali_osk.h>
+#include <uk/mali_ukk.h>
+
+mali_error ukk_session_init(ukk_session *ukk_session, ukk_dispatch_function dispatch, u16 version_major, u16 version_minor)
+{
+ OSK_ASSERT(NULL != ukk_session);
+ OSK_ASSERT(NULL != dispatch);
+
+ /* OS independent initialization of UKK context */
+ ukk_session->dispatch = dispatch;
+ ukk_session->version_major = version_major;
+ ukk_session->version_minor = version_minor;
+
+ /* OS specific initialization of UKK context */
+ ukk_session->internal_session.dummy = 0;
+ return MALI_ERROR_NONE;
+}
+
+void ukk_session_term(ukk_session *ukk_session)
+{
+ OSK_ASSERT(NULL != ukk_session);
+}
+
+mali_error ukk_copy_from_user( size_t bytes, void * kernel_buffer, const void * const user_buffer )
+{
+ if ( copy_from_user( kernel_buffer, user_buffer, bytes ) )
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+ return MALI_ERROR_NONE;
+}
+
+mali_error ukk_copy_to_user( size_t bytes, void * user_buffer, const void * const kernel_buffer )
+{
+ if ( copy_to_user( user_buffer, kernel_buffer, bytes ) )
+ {
+ return MALI_ERROR_FUNCTION_FAILED;
+ }
+ return MALI_ERROR_NONE;
+}
+
+static int __init ukk_module_init(void)
+{
+ if (MALI_ERROR_NONE != ukk_start())
+ {
+ return -EINVAL;
+ }
+ return 0;
+}
+
+static void __exit ukk_module_exit(void)
+{
+ ukk_stop();
+}
+
+EXPORT_SYMBOL(ukk_copy_from_user);
+EXPORT_SYMBOL(ukk_copy_to_user);
+EXPORT_SYMBOL(ukk_session_init);
+EXPORT_SYMBOL(ukk_session_term);
+EXPORT_SYMBOL(ukk_session_get);
+EXPORT_SYMBOL(ukk_call_prepare);
+EXPORT_SYMBOL(ukk_dispatch);
+
+module_init(ukk_module_init);
+module_exit(ukk_module_exit);
+
+#if MALI_LICENSE_IS_GPL || MALI_UNIT_TEST /* See MIDBASE-1204 */
+MODULE_LICENSE("GPL");
+#else
+MODULE_LICENSE("Proprietary");
+#endif
+MODULE_AUTHOR("ARM Ltd.");
+MODULE_VERSION("0.0");
--- /dev/null
+# Copyright:
+# ----------------------------------------------------------------------------
+# This confidential and proprietary software may be used only as authorized
+# by a licensing agreement from ARM Limited.
+# (C) COPYRIGHT 2010-2011 ARM Limited, ALL RIGHTS RESERVED
+# The entire notice above must be reproduced on all authorized copies and
+# copies may only be made to the extent permitted by a licensing agreement
+# from ARM Limited.
+# ----------------------------------------------------------------------------
+#
+
+import os
+Import('env')
+
+# Clone the environment so changes don't affect other build files
+env_ukk = env.Clone()
+
+# Source files required for the UKK.
+ukk_src = [Glob('*.c'), Glob('#uk/src/ukk/*.c')]
+
+if env_ukk['backend'] == 'kernel':
+ if env_ukk['v'] != '1':
+ env_ukk['MAKECOMSTR'] = '[MAKE] ${SOURCE.dir}'
+
+ # Note: cleaning via the Linux kernel build system does not yet work
+ if env_ukk.GetOption('clean') :
+ makeAction=Action("cd ${SOURCE.dir} && make clean", '$MAKECOMSTR')
+ else:
+ makeAction=Action("cd ${SOURCE.dir} && make PLATFORM=${platform} MALI_DEBUG=${debug} MALI_BACKEND_KERNEL=1 MALI_HW_VERSION=${hwver} MALI_BASE_TRACK_MEMLEAK=${base_qa} MALI_UNIT_TEST=${unit} MALI_LICENSE_IS_GPL=${mali_license_is_gpl} && cp ukk.ko $STATIC_LIB_PATH/ukk.ko", '$MAKECOMSTR')
+
+ # The target is ukk.ko, built from the source in ukk_src, via the action makeAction
+ # ukk.ko will be copied to $STATIC_LIB_PATH after being built by the standard Linux
+ # kernel build system, after which it can be installed to the directory specified if
+ # "libs_install" is set; this is done by LibTarget.
+ cmd = env_ukk.Command('$STATIC_LIB_PATH/ukk.ko', ukk_src, [makeAction])
+
+ env.Depends('$STATIC_LIB_PATH/ukk.ko', '$STATIC_LIB_PATH/libosk.a')
+
+ # Until we fathom out how the invoke the Linux build system to clean, we can use Clean
+ # to remove generated files.
+
+ patterns = ['*.mod.c', '*.o', '*.ko', '*.a', '.*.cmd', 'modules.order', '.tmp_versions', 'Module.symvers']
+
+ for p in patterns:
+ Clean(cmd, Glob('#uk/src/ukk/linux/%s' % p))
+ Clean(cmd, Glob('#uk/src/ukk/%s' % p))
+
+ env_ukk.ProgTarget('uk', cmd)
--- /dev/null
+/*
+ * This confidential and proprietary software may be used only as
+ * authorised by a licensing agreement from ARM Limited
+ * (C) COPYRIGHT 2010-2012 ARM Limited
+ * ALL RIGHTS RESERVED
+ * The entire notice above must be reproduced on all authorised
+ * copies and copies may only be made to the extent permitted
+ * by a licensing agreement from ARM Limited.
+ */
+
+#include <osk/mali_osk.h>
+#include <uk/mali_ukk.h>
+#include <plat/mali_ukk_os.h>
+
+mali_error ukk_start(void)
+{
+ return MALI_ERROR_NONE;
+}
+
+void ukk_stop(void)
+{
+}
+
+void ukk_call_prepare(ukk_call_context * const ukk_ctx, ukk_session * const session)
+{
+ OSK_ASSERT(NULL != ukk_ctx);
+ OSK_ASSERT(NULL != session);
+
+ ukk_ctx->ukk_session = session;
+}
+
+void *ukk_session_get(ukk_call_context * const ukk_ctx)
+{
+ OSK_ASSERT(NULL != ukk_ctx);
+ return ukk_ctx->ukk_session;
+}
+
+static mali_error ukkp_dispatch_call(ukk_call_context *ukk_ctx, void *args, u32 args_size)
+{
+ uk_header *header = (uk_header *)args;
+ mali_error ret = MALI_ERROR_NONE;
+
+ if(UKP_FUNC_ID_CHECK_VERSION == header->id)
+ {
+ if (args_size == sizeof(uku_version_check_args))
+ {
+ ukk_session *ukk_session = ukk_session_get(ukk_ctx);
+ uku_version_check_args *version_check = (uku_version_check_args *)args;
+
+ version_check->major = ukk_session->version_major;
+ version_check->minor = ukk_session->version_minor;
+ header->ret = MALI_ERROR_NONE;
+ }
+ else
+ {
+ header->ret = MALI_ERROR_FUNCTION_FAILED;
+ }
+ }
+ else
+ {
+ ret = MALI_ERROR_FUNCTION_FAILED; /* not handled */
+ }
+ return ret;
+}
+
+mali_error ukk_dispatch(ukk_call_context * const ukk_ctx, void * const args, u32 args_size)
+{
+ mali_error ret;
+ uk_header *header = (uk_header *)args;
+
+ OSK_ASSERT(NULL != ukk_ctx);
+ OSK_ASSERT(NULL != args);
+
+ /* Verify args_size both in debug and release builds */
+ OSK_ASSERT(args_size >= sizeof(uk_header));
+ if (args_size < sizeof(uk_header)) return MALI_ERROR_FUNCTION_FAILED;
+
+ if (header->id >= UK_FUNC_ID)
+ {
+ ret = ukk_ctx->ukk_session->dispatch(ukk_ctx, args, args_size);
+ }
+ else
+ {
+ ret = ukkp_dispatch_call(ukk_ctx, args, args_size);
+ }
+ return ret;
+}
--- /dev/null
+# This confidential and proprietary software may be used only as
+# authorised by a licensing agreement from ARM Limited
+# (C) COPYRIGHT 2010-2011 ARM Limited
+# ALL RIGHTS RESERVED
+# The entire notice above must be reproduced on all authorised
+# copies and copies may only be made to the extent permitted
+# by a licensing agreement from ARM Limited.
+
+Import( 'env' )
+
+
+if env['backend'] == 'kernel':
+ SConscript(env['kernel'] + '/sconscript')
+else:
+ SConscript('userspace/sconscript')
space through the SMMU (System Memory Management Unit)
hardware included on Tegra SoCs.
+config EXYNOS_IOMMU
+ bool "Exynos IOMMU Support"
+ depends on EXYNOS_DEV_SYSMMU
+ select IOMMU_API
+ select ARM_DMA_USE_IOMMU
+ help
+ Support for the IOMMU(System MMU) of Samsung Exynos application
+ processor family. This enables H/W multimedia accellerators to see
+ non-linear physical memory chunks as a linear memory in their
+ address spaces
+
+ If unsure, say N here.
+
+config EXYNOS_IOMMU_DEBUG
+ bool "Debugging log for Exynos IOMMU"
+ depends on EXYNOS_IOMMU
+ help
+ Select this to see the detailed log message that shows what
+ happens in the IOMMU driver
+
+ Say N unless you need kernel log message for IOMMU debugging
+
endif # IOMMU_SUPPORT
obj-$(CONFIG_OMAP_IOMMU_DEBUG) += omap-iommu-debug.o
obj-$(CONFIG_TEGRA_IOMMU_GART) += tegra-gart.o
obj-$(CONFIG_TEGRA_IOMMU_SMMU) += tegra-smmu.o
+obj-$(CONFIG_EXYNOS_IOMMU) += exynos-iommu.o
--- /dev/null
+/* linux/drivers/iommu/exynos_iommu.c
+ *
+ * Copyright (c) 2011 Samsung Electronics Co., Ltd.
+ * http://www.samsung.com
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifdef CONFIG_EXYNOS_IOMMU_DEBUG
+#define DEBUG
+#endif
+
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/io.h>
+#include <linux/interrupt.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/pm_runtime.h>
+#include <linux/clk.h>
+#include <linux/err.h>
+#include <linux/mm.h>
+#include <linux/iommu.h>
+#include <linux/errno.h>
+#include <linux/list.h>
+#include <linux/memblock.h>
+#include <linux/export.h>
+
+#include <asm/cacheflush.h>
+#include <asm/pgtable.h>
+
+#include <mach/sysmmu.h>
+
+/* We does not consider super section mapping (16MB) */
+#define SECT_ORDER 20
+#define LPAGE_ORDER 16
+#define SPAGE_ORDER 12
+
+#define SECT_SIZE (1 << SECT_ORDER)
+#define LPAGE_SIZE (1 << LPAGE_ORDER)
+#define SPAGE_SIZE (1 << SPAGE_ORDER)
+
+#define SECT_MASK (~(SECT_SIZE - 1))
+#define LPAGE_MASK (~(LPAGE_SIZE - 1))
+#define SPAGE_MASK (~(SPAGE_SIZE - 1))
+
+#define lv1ent_fault(sent) (((*(sent) & 3) == 0) || ((*(sent) & 3) == 3))
+#define lv1ent_page(sent) ((*(sent) & 3) == 1)
+#define lv1ent_section(sent) ((*(sent) & 3) == 2)
+
+#define lv2ent_fault(pent) ((*(pent) & 3) == 0)
+#define lv2ent_small(pent) ((*(pent) & 2) == 2)
+#define lv2ent_large(pent) ((*(pent) & 3) == 1)
+
+#define section_phys(sent) (*(sent) & SECT_MASK)
+#define section_offs(iova) ((iova) & 0xFFFFF)
+#define lpage_phys(pent) (*(pent) & LPAGE_MASK)
+#define lpage_offs(iova) ((iova) & 0xFFFF)
+#define spage_phys(pent) (*(pent) & SPAGE_MASK)
+#define spage_offs(iova) ((iova) & 0xFFF)
+
+#define lv1ent_offset(iova) ((iova) >> SECT_ORDER)
+#define lv2ent_offset(iova) (((iova) & 0xFF000) >> SPAGE_ORDER)
+
+#define NUM_LV1ENTRIES 4096
+#define NUM_LV2ENTRIES 256
+
+#define LV2TABLE_SIZE (NUM_LV2ENTRIES * sizeof(long))
+
+#define SPAGES_PER_LPAGE (LPAGE_SIZE / SPAGE_SIZE)
+
+#define lv2table_base(sent) (*(sent) & 0xFFFFFC00)
+
+#define mk_lv1ent_sect(pa) ((pa) | 2)
+#define mk_lv1ent_page(pa) ((pa) | 1)
+#define mk_lv2ent_lpage(pa) ((pa) | 1)
+#define mk_lv2ent_spage(pa) ((pa) | 2)
+
+#define CTRL_ENABLE 0x5
+#define CTRL_BLOCK 0x7
+#define CTRL_DISABLE 0x0
+
+#define REG_MMU_CTRL 0x000
+#define REG_MMU_CFG 0x004
+#define REG_MMU_STATUS 0x008
+#define REG_MMU_FLUSH 0x00C
+#define REG_MMU_FLUSH_ENTRY 0x010
+#define REG_PT_BASE_ADDR 0x014
+#define REG_INT_STATUS 0x018
+#define REG_INT_CLEAR 0x01C
+
+#define REG_PAGE_FAULT_ADDR 0x024
+#define REG_AW_FAULT_ADDR 0x028
+#define REG_AR_FAULT_ADDR 0x02C
+#define REG_DEFAULT_SLAVE_ADDR 0x030
+
+#define REG_MMU_VERSION 0x034
+
+#define REG_PB0_SADDR 0x04C
+#define REG_PB0_EADDR 0x050
+#define REG_PB1_SADDR 0x054
+#define REG_PB1_EADDR 0x058
+
+static unsigned long *section_entry(unsigned long *pgtable, unsigned long iova)
+{
+ return pgtable + lv1ent_offset(iova);
+}
+
+static unsigned long *page_entry(unsigned long *sent, unsigned long iova)
+{
+ return (unsigned long *)__va(lv2table_base(sent)) + lv2ent_offset(iova);
+}
+
+enum exynos_sysmmu_inttype {
+ SYSMMU_PAGEFAULT,
+ SYSMMU_AR_MULTIHIT,
+ SYSMMU_AW_MULTIHIT,
+ SYSMMU_BUSERROR,
+ SYSMMU_AR_SECURITY,
+ SYSMMU_AR_ACCESS,
+ SYSMMU_AW_SECURITY,
+ SYSMMU_AW_PROTECTION, /* 7 */
+ SYSMMU_FAULT_UNKNOWN,
+ SYSMMU_FAULTS_NUM
+};
+
+typedef int (*sysmmu_fault_handler_t)(enum exynos_sysmmu_inttype itype,
+ unsigned long pgtable_base, unsigned long fault_addr);
+
+static unsigned short fault_reg_offset[SYSMMU_FAULTS_NUM] = {
+ REG_PAGE_FAULT_ADDR,
+ REG_AR_FAULT_ADDR,
+ REG_AW_FAULT_ADDR,
+ REG_DEFAULT_SLAVE_ADDR,
+ REG_AR_FAULT_ADDR,
+ REG_AR_FAULT_ADDR,
+ REG_AW_FAULT_ADDR,
+ REG_AW_FAULT_ADDR
+};
+
+static char *sysmmu_fault_name[SYSMMU_FAULTS_NUM] = {
+ "PAGE FAULT",
+ "AR MULTI-HIT FAULT",
+ "AW MULTI-HIT FAULT",
+ "BUS ERROR",
+ "AR SECURITY PROTECTION FAULT",
+ "AR ACCESS PROTECTION FAULT",
+ "AW SECURITY PROTECTION FAULT",
+ "AW ACCESS PROTECTION FAULT",
+ "UNKNOWN FAULT"
+};
+
+struct exynos_iommu_domain {
+ struct list_head clients; /* list of sysmmu_drvdata.node */
+ unsigned long *pgtable; /* lv1 page table, 16KB */
+ short *lv2entcnt; /* free lv2 entry counter for each section */
+ spinlock_t lock; /* lock for this structure */
+ spinlock_t pgtablelock; /* lock for modifying page table @ pgtable */
+};
+
+struct sysmmu_drvdata {
+ struct list_head node; /* entry of exynos_iommu_domain.clients */
+ struct device *sysmmu; /* System MMU's device descriptor */
+ struct device *dev; /* Owner of system MMU */
+ char *dbgname;
+ int nsfrs;
+ void __iomem **sfrbases;
+ struct clk *clk[2];
+ int activations;
+ rwlock_t lock;
+ struct iommu_domain *domain;
+ sysmmu_fault_handler_t fault_handler;
+ unsigned long pgtable;
+};
+
+static bool set_sysmmu_active(struct sysmmu_drvdata *data)
+{
+ /* return true if the System MMU was not active previously
+ and it needs to be initialized */
+ return ++data->activations == 1;
+}
+
+static bool set_sysmmu_inactive(struct sysmmu_drvdata *data)
+{
+ /* return true if the System MMU is needed to be disabled */
+ BUG_ON(data->activations < 1);
+ return --data->activations == 0;
+}
+
+static bool is_sysmmu_active(struct sysmmu_drvdata *data)
+{
+ return data->activations > 0;
+}
+
+static void sysmmu_block(void __iomem *sfrbase)
+{
+ __raw_writel(CTRL_BLOCK, sfrbase + REG_MMU_CTRL);
+}
+
+static void sysmmu_unblock(void __iomem *sfrbase)
+{
+ __raw_writel(CTRL_ENABLE, sfrbase + REG_MMU_CTRL);
+}
+
+static void __sysmmu_tlb_invalidate(void __iomem *sfrbase)
+{
+ __raw_writel(0x1, sfrbase + REG_MMU_FLUSH);
+}
+
+static void __sysmmu_tlb_invalidate_entry(void __iomem *sfrbase,
+ unsigned long iova)
+{
+ __raw_writel((iova & SPAGE_MASK) | 1, sfrbase + REG_MMU_FLUSH_ENTRY);
+}
+
+static void __sysmmu_set_ptbase(void __iomem *sfrbase,
+ unsigned long pgd)
+{
+ __raw_writel(0x1, sfrbase + REG_MMU_CFG); /* 16KB LV1, LRU */
+ __raw_writel(pgd, sfrbase + REG_PT_BASE_ADDR);
+
+ __sysmmu_tlb_invalidate(sfrbase);
+}
+
+static void __sysmmu_set_prefbuf(void __iomem *sfrbase, unsigned long base,
+ unsigned long size, int idx)
+{
+ __raw_writel(base, sfrbase + REG_PB0_SADDR + idx * 8);
+ __raw_writel(size - 1 + base, sfrbase + REG_PB0_EADDR + idx * 8);
+}
+
+void exynos_sysmmu_set_prefbuf(struct device *dev,
+ unsigned long base0, unsigned long size0,
+ unsigned long base1, unsigned long size1)
+{
+ struct sysmmu_drvdata *data = dev_get_drvdata(dev->archdata.iommu);
+ unsigned long flags;
+ int i;
+
+ BUG_ON((base0 + size0) <= base0);
+ BUG_ON((size1 > 0) && ((base1 + size1) <= base1));
+
+ read_lock_irqsave(&data->lock, flags);
+ if (!is_sysmmu_active(data))
+ goto finish;
+
+ for (i = 0; i < data->nsfrs; i++) {
+ if ((readl(data->sfrbases[i] + REG_MMU_VERSION) >> 28) == 3) {
+ sysmmu_block(data->sfrbases[i]);
+
+ if (size1 == 0) {
+ if (size0 <= SZ_128K) {
+ base1 = base0;
+ size1 = size0;
+ } else {
+ size1 = size0 -
+ ALIGN(size0 / 2, SZ_64K);
+ size0 = size0 - size1;
+ base1 = base0 + size0;
+ }
+ }
+
+ __sysmmu_set_prefbuf(
+ data->sfrbases[i], base0, size0, 0);
+ __sysmmu_set_prefbuf(
+ data->sfrbases[i], base1, size1, 1);
+
+ sysmmu_unblock(data->sfrbases[i]);
+ }
+ }
+finish:
+ read_unlock_irqrestore(&data->lock, flags);
+}
+
+static void __set_fault_handler(struct sysmmu_drvdata *data,
+ sysmmu_fault_handler_t handler)
+{
+ unsigned long flags;
+
+ write_lock_irqsave(&data->lock, flags);
+ data->fault_handler = handler;
+ write_unlock_irqrestore(&data->lock, flags);
+}
+
+void exynos_sysmmu_set_fault_handler(struct device *dev,
+ sysmmu_fault_handler_t handler)
+{
+ struct sysmmu_drvdata *data = dev_get_drvdata(dev->archdata.iommu);
+
+ __set_fault_handler(data, handler);
+}
+
+static int default_fault_handler(enum exynos_sysmmu_inttype itype,
+ unsigned long pgtable_base, unsigned long fault_addr)
+{
+ unsigned long *ent;
+
+ if ((itype >= SYSMMU_FAULTS_NUM) || (itype < SYSMMU_PAGEFAULT))
+ itype = SYSMMU_FAULT_UNKNOWN;
+
+ pr_err("%s occured at 0x%lx(Page table base: 0x%lx)\n",
+ sysmmu_fault_name[itype], fault_addr, pgtable_base);
+
+ ent = section_entry(__va(pgtable_base), fault_addr);
+ pr_err("\tLv1 entry: 0x%lx\n", *ent);
+
+ if (lv1ent_page(ent)) {
+ ent = page_entry(ent, fault_addr);
+ pr_err("\t Lv2 entry: 0x%lx\n", *ent);
+ }
+
+ pr_err("Generating Kernel OOPS... because it is unrecoverable.\n");
+
+ BUG();
+
+ return 0;
+}
+
+static irqreturn_t exynos_sysmmu_irq(int irq, void *dev_id)
+{
+ /* SYSMMU is in blocked when interrupt occurred. */
+ struct sysmmu_drvdata *data = dev_id;
+ struct resource *irqres;
+ struct platform_device *pdev;
+ enum exynos_sysmmu_inttype itype;
+ unsigned long addr = -1;
+
+ int i, ret = -ENOSYS;
+
+ read_lock(&data->lock);
+
+ WARN_ON(!is_sysmmu_active(data));
+
+ pdev = to_platform_device(data->sysmmu);
+ for (i = 0; i < (pdev->num_resources / 2); i++) {
+ irqres = platform_get_resource(pdev, IORESOURCE_IRQ, i);
+ if (irqres && ((int)irqres->start == irq))
+ break;
+ }
+
+ if (i == pdev->num_resources) {
+ itype = SYSMMU_FAULT_UNKNOWN;
+ } else {
+ itype = (enum exynos_sysmmu_inttype)
+ __ffs(__raw_readl(data->sfrbases[i] + REG_INT_STATUS));
+ if (WARN_ON(!((itype >= 0) && (itype < SYSMMU_FAULT_UNKNOWN))))
+ itype = SYSMMU_FAULT_UNKNOWN;
+ else
+ addr = __raw_readl(
+ data->sfrbases[i] + fault_reg_offset[itype]);
+ }
+
+ if (data->domain)
+ ret = report_iommu_fault(data->domain, data->dev,
+ addr, itype);
+
+ if ((ret == -ENOSYS) && data->fault_handler) {
+ unsigned long base = data->pgtable;
+ if (itype != SYSMMU_FAULT_UNKNOWN)
+ base = __raw_readl(
+ data->sfrbases[i] + REG_PT_BASE_ADDR);
+ ret = data->fault_handler(itype, base, addr);
+ }
+
+ if (!ret && (itype != SYSMMU_FAULT_UNKNOWN))
+ __raw_writel(1 << itype, data->sfrbases[i] + REG_INT_CLEAR);
+ else
+ dev_dbg(data->sysmmu, "(%s) %s is not handled.\n",
+ data->dbgname, sysmmu_fault_name[itype]);
+
+ if (itype != SYSMMU_FAULT_UNKNOWN)
+ sysmmu_unblock(data->sfrbases[i]);
+
+ read_unlock(&data->lock);
+
+ return IRQ_HANDLED;
+}
+
+static bool __exynos_sysmmu_disable(struct sysmmu_drvdata *data)
+{
+ unsigned long flags;
+ bool disabled = false;
+ int i;
+
+ write_lock_irqsave(&data->lock, flags);
+
+ if (!set_sysmmu_inactive(data))
+ goto finish;
+
+ for (i = 0; i < data->nsfrs; i++)
+ __raw_writel(CTRL_DISABLE, data->sfrbases[i] + REG_MMU_CTRL);
+
+ if (data->clk[1])
+ clk_disable(data->clk[1]);
+ if (data->clk[0])
+ clk_disable(data->clk[0]);
+
+ disabled = true;
+ data->pgtable = 0;
+ data->domain = NULL;
+finish:
+ write_unlock_irqrestore(&data->lock, flags);
+
+ if (disabled)
+ dev_dbg(data->sysmmu, "(%s) Disabled\n", data->dbgname);
+ else
+ dev_dbg(data->sysmmu, "(%s) %d times left to be disabled\n",
+ data->dbgname, data->activations);
+
+ return disabled;
+}
+
+/* __exynos_sysmmu_enable: Enables System MMU
+ *
+ * returns -error if an error occurred and System MMU is not enabled,
+ * 0 if the System MMU has been just enabled and 1 if System MMU was already
+ * enabled before.
+ */
+static int __exynos_sysmmu_enable(struct sysmmu_drvdata *data,
+ unsigned long pgtable, struct iommu_domain *domain)
+{
+ int i, ret;
+ unsigned long flags;
+
+ write_lock_irqsave(&data->lock, flags);
+
+ if (!set_sysmmu_active(data)) {
+ if (WARN_ON(pgtable != data->pgtable)) {
+ ret = -EBUSY;
+ set_sysmmu_inactive(data);
+ } else {
+ ret = 1;
+ }
+
+ dev_dbg(data->sysmmu, "(%s) Already enabled\n", data->dbgname);
+ goto finish;
+ }
+
+ ret = 0;
+
+ if (data->clk[0])
+ clk_enable(data->clk[0]);
+ if (data->clk[1])
+ clk_enable(data->clk[1]);
+
+ data->pgtable = pgtable;
+
+ for (i = 0; i < data->nsfrs; i++) {
+ __sysmmu_set_ptbase(data->sfrbases[i], pgtable);
+
+ if ((readl(data->sfrbases[i] + REG_MMU_VERSION) >> 28) == 3) {
+ /* System MMU version is 3.x */
+ __raw_writel((1 << 12) | (2 << 28),
+ data->sfrbases[i] + REG_MMU_CFG);
+ __sysmmu_set_prefbuf(data->sfrbases[i], 0, -1, 0);
+ __sysmmu_set_prefbuf(data->sfrbases[i], 0, -1, 1);
+ }
+
+ __raw_writel(CTRL_ENABLE, data->sfrbases[i] + REG_MMU_CTRL);
+ }
+
+ data->domain = domain;
+
+ dev_dbg(data->sysmmu, "(%s) Enabled\n", data->dbgname);
+finish:
+ write_unlock_irqrestore(&data->lock, flags);
+
+ if ((ret < 0) && (ret != -EBUSY)) {
+ __exynos_sysmmu_disable(data);
+ dev_dbg(data->sysmmu, "(%s) Failed to enable\n", data->dbgname);
+ }
+
+ return ret;
+}
+
+int exynos_sysmmu_enable(struct device *dev, unsigned long pgtable)
+{
+ struct sysmmu_drvdata *data = dev_get_drvdata(dev->archdata.iommu);
+ int ret;
+
+ BUG_ON(!memblock_is_memory(pgtable));
+
+ ret = pm_runtime_get_sync(data->sysmmu);
+ if (ret < 0)
+ return ret;
+
+ ret = __exynos_sysmmu_enable(data, pgtable, NULL);
+ if (ret < 0)
+ pm_runtime_put(data->sysmmu);
+ else
+ data->dev = dev;
+
+ return ret;
+}
+
+bool exynos_sysmmu_disable(struct device *dev)
+{
+ struct sysmmu_drvdata *data = dev_get_drvdata(dev->archdata.iommu);
+ bool disabled;
+
+ disabled = __exynos_sysmmu_disable(data);
+ pm_runtime_put(data->sysmmu);
+
+ return disabled;
+}
+
+static void sysmmu_tlb_invalidate_entry(struct device *dev, unsigned long iova)
+{
+ unsigned long flags;
+ struct sysmmu_drvdata *data = dev_get_drvdata(dev->archdata.iommu);
+
+ read_lock_irqsave(&data->lock, flags);
+
+ if (is_sysmmu_active(data)) {
+ int i;
+ for (i = 0; i < data->nsfrs; i++) {
+ sysmmu_block(data->sfrbases[i]);
+ __sysmmu_tlb_invalidate_entry(data->sfrbases[i], iova);
+ sysmmu_unblock(data->sfrbases[i]);
+ }
+ } else {
+ dev_dbg(data->sysmmu,
+ "(%s) Disabled. Skipping invalidating TLB.\n",
+ data->dbgname);
+ }
+
+ read_unlock_irqrestore(&data->lock, flags);
+}
+
+void exynos_sysmmu_tlb_invalidate(struct device *dev)
+{
+ unsigned long flags;
+ struct sysmmu_drvdata *data = dev_get_drvdata(dev->archdata.iommu);
+
+ read_lock_irqsave(&data->lock, flags);
+
+ if (is_sysmmu_active(data)) {
+ int i;
+ for (i = 0; i < data->nsfrs; i++) {
+ sysmmu_block(data->sfrbases[i]);
+ __sysmmu_tlb_invalidate(data->sfrbases[i]);
+ sysmmu_unblock(data->sfrbases[i]);
+ }
+ } else {
+ dev_dbg(data->sysmmu,
+ "(%s) Disabled. Skipping invalidating TLB.\n",
+ data->dbgname);
+ }
+
+ read_unlock_irqrestore(&data->lock, flags);
+}
+
+static int exynos_sysmmu_probe(struct platform_device *pdev)
+{
+ int i, ret;
+ struct device *dev;
+ struct sysmmu_drvdata *data;
+
+ dev = &pdev->dev;
+
+ data = kzalloc(sizeof(*data), GFP_KERNEL);
+ if (!data) {
+ dev_dbg(dev, "Not enough memory\n");
+ ret = -ENOMEM;
+ goto err_alloc;
+ }
+
+ ret = dev_set_drvdata(dev, data);
+ if (ret) {
+ dev_dbg(dev, "Unabled to initialize driver data\n");
+ goto err_init;
+ }
+
+ data->nsfrs = pdev->num_resources / 2;
+ data->sfrbases = kmalloc(sizeof(*data->sfrbases) * data->nsfrs,
+ GFP_KERNEL);
+ if (data->sfrbases == NULL) {
+ dev_dbg(dev, "Not enough memory\n");
+ ret = -ENOMEM;
+ goto err_init;
+ }
+
+ for (i = 0; i < data->nsfrs; i++) {
+ struct resource *res;
+ res = platform_get_resource(pdev, IORESOURCE_MEM, i);
+ if (!res) {
+ dev_dbg(dev, "Unable to find IOMEM region\n");
+ ret = -ENOENT;
+ goto err_res;
+ }
+
+ data->sfrbases[i] = ioremap(res->start, resource_size(res));
+ if (!data->sfrbases[i]) {
+ dev_dbg(dev, "Unable to map IOMEM @ PA:%#x\n",
+ res->start);
+ ret = -ENOENT;
+ goto err_res;
+ }
+ }
+
+ for (i = 0; i < data->nsfrs; i++) {
+ ret = platform_get_irq(pdev, i);
+ if (ret <= 0) {
+ dev_dbg(dev, "Unable to find IRQ resource\n");
+ goto err_irq;
+ }
+
+ ret = request_irq(ret, exynos_sysmmu_irq, 0,
+ dev_name(dev), data);
+ if (ret) {
+ dev_dbg(dev, "Unabled to register interrupt handler\n");
+ goto err_irq;
+ }
+ }
+
+ if (dev_get_platdata(dev)) {
+ char *deli, *beg;
+ struct sysmmu_platform_data *platdata = dev_get_platdata(dev);
+
+ beg = platdata->clockname;
+
+ for (deli = beg; (*deli != '\0') && (*deli != ','); deli++)
+ /* NOTHING */;
+
+ if (*deli == '\0')
+ deli = NULL;
+ else
+ *deli = '\0';
+
+ data->clk[0] = clk_get(dev, beg);
+ if (IS_ERR(data->clk[0])) {
+ data->clk[0] = NULL;
+ dev_dbg(dev, "No clock descriptor registered\n");
+ }
+
+ if (data->clk[0] && deli) {
+ *deli = ',';
+ data->clk[1] = clk_get(dev, deli + 1);
+ if (IS_ERR(data->clk[1]))
+ data->clk[1] = NULL;
+ }
+
+ data->dbgname = platdata->dbgname;
+ }
+
+ data->sysmmu = dev;
+ rwlock_init(&data->lock);
+ INIT_LIST_HEAD(&data->node);
+
+ __set_fault_handler(data, &default_fault_handler);
+
+ pm_runtime_enable(dev);
+
+ dev_dbg(dev, "(%s) Initialized\n", data->dbgname);
+ return 0;
+err_irq:
+ while (i-- > 0) {
+ int irq;
+
+ irq = platform_get_irq(pdev, i);
+ free_irq(irq, data);
+ }
+err_res:
+ while (data->nsfrs-- > 0)
+ iounmap(data->sfrbases[data->nsfrs]);
+ kfree(data->sfrbases);
+err_init:
+ kfree(data);
+err_alloc:
+ dev_err(dev, "Failed to initialize\n");
+ return ret;
+}
+
+static int exynos_pm_resume(struct device *dev)
+{
+ struct sysmmu_drvdata *data;
+
+ data = dev_get_drvdata(dev);
+
+ if (is_sysmmu_active(data))
+ __exynos_sysmmu_enable(data, data->pgtable, NULL);
+
+ return 0;
+}
+
+const struct dev_pm_ops exynos_pm_ops = {
+ .resume = &exynos_pm_resume,
+};
+
+#ifdef CONFIG_OF
+static const struct of_device_id sysmmu_of_match[] = {
+ { .compatible = "samsung,s5p-sysmmu" },
+ {},
+};
+MODULE_DEVICE_TABLE(of, sysmmu_of_match);
+#else
+#define sysmmu_of_match NULL
+#endif
+
+static struct platform_driver exynos_sysmmu_driver = {
+ .probe = exynos_sysmmu_probe,
+ .driver = {
+ .owner = THIS_MODULE,
+ .name = "s5p-sysmmu",
+ .pm = &exynos_pm_ops,
+ .of_match_table = sysmmu_of_match,
+ }
+};
+
+static inline void pgtable_flush(void *vastart, void *vaend)
+{
+ dmac_flush_range(vastart, vaend);
+ outer_flush_range(virt_to_phys(vastart),
+ virt_to_phys(vaend));
+}
+
+static int exynos_iommu_domain_init(struct iommu_domain *domain)
+{
+ struct exynos_iommu_domain *priv;
+
+ priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ priv->pgtable = (unsigned long *)__get_free_pages(
+ GFP_KERNEL | __GFP_ZERO, 2);
+ if (!priv->pgtable)
+ goto err_pgtable;
+
+ priv->lv2entcnt = (short *)__get_free_pages(
+ GFP_KERNEL | __GFP_ZERO, 1);
+ if (!priv->lv2entcnt)
+ goto err_counter;
+
+ pgtable_flush(priv->pgtable, priv->pgtable + NUM_LV1ENTRIES);
+
+ spin_lock_init(&priv->lock);
+ spin_lock_init(&priv->pgtablelock);
+ INIT_LIST_HEAD(&priv->clients);
+
+ domain->priv = priv;
+ return 0;
+
+err_counter:
+ free_pages((unsigned long)priv->pgtable, 2);
+err_pgtable:
+ kfree(priv);
+ return -ENOMEM;
+}
+
+static void exynos_iommu_domain_destroy(struct iommu_domain *domain)
+{
+ struct exynos_iommu_domain *priv = domain->priv;
+ struct sysmmu_drvdata *data;
+ unsigned long flags;
+ int i;
+
+ WARN_ON(!list_empty(&priv->clients));
+
+ spin_lock_irqsave(&priv->lock, flags);
+
+ list_for_each_entry(data, &priv->clients, node) {
+ while (!exynos_sysmmu_disable(data->dev))
+ ; /* until System MMU is actually disabled */
+ }
+
+ spin_unlock_irqrestore(&priv->lock, flags);
+
+ for (i = 0; i < NUM_LV1ENTRIES; i++)
+ if (lv1ent_page(priv->pgtable + i))
+ kfree(__va(lv2table_base(priv->pgtable + i)));
+
+ free_pages((unsigned long)priv->pgtable, 2);
+ free_pages((unsigned long)priv->lv2entcnt, 1);
+ kfree(domain->priv);
+ domain->priv = NULL;
+}
+
+static int exynos_iommu_attach_device(struct iommu_domain *domain,
+ struct device *dev)
+{
+ struct sysmmu_drvdata *data = dev_get_drvdata(dev->archdata.iommu);
+ struct exynos_iommu_domain *priv = domain->priv;
+ unsigned long flags;
+ int ret;
+
+ /* If there is no SYSMMU, this could be a virtual device with a common
+ * IOMMU mapping shared with another device e.g. DRM with DRM-FIMD
+ */
+ if (data==NULL) {
+ dev_err(dev, "No SYSMMU found\n");
+ return 0;
+ }
+
+ ret = pm_runtime_get_sync(data->sysmmu);
+ if (ret < 0)
+ return ret;
+
+ ret = 0;
+
+ spin_lock_irqsave(&priv->lock, flags);
+
+ ret = __exynos_sysmmu_enable(data, __pa(priv->pgtable), domain);
+
+ if (ret == 0) {
+ /* 'data->node' must not appear in priv->clients */
+ BUG_ON(!list_empty(&data->node));
+ data->dev = dev;
+ list_add_tail(&data->node, &priv->clients);
+ }
+
+ spin_unlock_irqrestore(&priv->lock, flags);
+
+ if (ret < 0) {
+ dev_err(dev, "%s: Failed to attach IOMMU with pgtable %#lx\n",
+ __func__, __pa(priv->pgtable));
+ pm_runtime_put(data->sysmmu);
+ } else if (ret > 0) {
+ dev_dbg(dev, "%s: IOMMU with pgtable 0x%lx already attached\n",
+ __func__, __pa(priv->pgtable));
+ } else {
+ dev_dbg(dev, "%s: Attached new IOMMU with pgtable 0x%lx\n",
+ __func__, __pa(priv->pgtable));
+ }
+
+ return ret;
+}
+
+static void exynos_iommu_detach_device(struct iommu_domain *domain,
+ struct device *dev)
+{
+ struct sysmmu_drvdata *data = dev_get_drvdata(dev->archdata.iommu);
+ struct exynos_iommu_domain *priv = domain->priv;
+ struct list_head *pos;
+ unsigned long flags;
+ bool found = false;
+
+ spin_lock_irqsave(&priv->lock, flags);
+
+ list_for_each(pos, &priv->clients) {
+ if (list_entry(pos, struct sysmmu_drvdata, node) == data) {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ goto finish;
+
+ if (__exynos_sysmmu_disable(data)) {
+ dev_dbg(dev, "%s: Detached IOMMU with pgtable %#lx\n",
+ __func__, __pa(priv->pgtable));
+ list_del(&data->node);
+ INIT_LIST_HEAD(&data->node);
+
+ } else {
+ dev_dbg(dev, "%s: Detaching IOMMU with pgtable %#lx delayed",
+ __func__, __pa(priv->pgtable));
+ }
+
+finish:
+ spin_unlock_irqrestore(&priv->lock, flags);
+
+ if (found)
+ pm_runtime_put(data->sysmmu);
+}
+
+static unsigned long *alloc_lv2entry(unsigned long *sent, unsigned long iova,
+ short *pgcounter)
+{
+ if (lv1ent_fault(sent)) {
+ unsigned long *pent;
+
+ pent = kzalloc(LV2TABLE_SIZE, GFP_ATOMIC);
+ BUG_ON((unsigned long)pent & (LV2TABLE_SIZE - 1));
+ if (!pent)
+ return NULL;
+
+ *sent = mk_lv1ent_page(__pa(pent));
+ *pgcounter = NUM_LV2ENTRIES;
+ pgtable_flush(pent, pent + NUM_LV2ENTRIES);
+ pgtable_flush(sent, sent + 1);
+ }
+
+ return page_entry(sent, iova);
+}
+
+static int lv1set_section(unsigned long *sent, phys_addr_t paddr, short *pgcnt)
+{
+ if (lv1ent_section(sent))
+ return -EADDRINUSE;
+
+ if (lv1ent_page(sent)) {
+ if (*pgcnt != NUM_LV2ENTRIES)
+ return -EADDRINUSE;
+
+ kfree(page_entry(sent, 0));
+
+ *pgcnt = 0;
+ }
+
+ *sent = mk_lv1ent_sect(paddr);
+
+ pgtable_flush(sent, sent + 1);
+
+ return 0;
+}
+
+static int lv2set_page(unsigned long *pent, phys_addr_t paddr, size_t size,
+ short *pgcnt)
+{
+ if (size == SPAGE_SIZE) {
+ if (!lv2ent_fault(pent))
+ return -EADDRINUSE;
+
+ *pent = mk_lv2ent_spage(paddr);
+ pgtable_flush(pent, pent + 1);
+ *pgcnt -= 1;
+ } else { /* size == LPAGE_SIZE */
+ int i;
+ for (i = 0; i < SPAGES_PER_LPAGE; i++, pent++) {
+ if (!lv2ent_fault(pent)) {
+ memset(pent, 0, sizeof(*pent) * i);
+ return -EADDRINUSE;
+ }
+
+ *pent = mk_lv2ent_lpage(paddr);
+ }
+ pgtable_flush(pent - SPAGES_PER_LPAGE, pent);
+ *pgcnt -= SPAGES_PER_LPAGE;
+ }
+
+ return 0;
+}
+
+static int exynos_iommu_map(struct iommu_domain *domain, unsigned long iova,
+ phys_addr_t paddr, size_t size, int prot)
+{
+ struct exynos_iommu_domain *priv = domain->priv;
+ unsigned long *entry;
+ unsigned long flags;
+ int ret = -ENOMEM;
+
+ BUG_ON(priv->pgtable == NULL);
+
+ spin_lock_irqsave(&priv->pgtablelock, flags);
+
+ entry = section_entry(priv->pgtable, iova);
+
+ if (size == SECT_SIZE) {
+ ret = lv1set_section(entry, paddr,
+ &priv->lv2entcnt[lv1ent_offset(iova)]);
+ } else {
+ unsigned long *pent;
+
+ pent = alloc_lv2entry(entry, iova,
+ &priv->lv2entcnt[lv1ent_offset(iova)]);
+
+ if (!pent)
+ ret = -ENOMEM;
+ else
+ ret = lv2set_page(pent, paddr, size,
+ &priv->lv2entcnt[lv1ent_offset(iova)]);
+ }
+
+ if (ret) {
+ pr_debug("%s: Failed to map iova 0x%lx/0x%x bytes\n",
+ __func__, iova, size);
+ }
+
+ spin_unlock_irqrestore(&priv->pgtablelock, flags);
+
+ return ret;
+}
+
+static size_t exynos_iommu_unmap(struct iommu_domain *domain,
+ unsigned long iova, size_t size)
+{
+ struct exynos_iommu_domain *priv = domain->priv;
+ struct sysmmu_drvdata *data;
+ unsigned long flags;
+ unsigned long *ent;
+
+ BUG_ON(priv->pgtable == NULL);
+
+ spin_lock_irqsave(&priv->pgtablelock, flags);
+
+ ent = section_entry(priv->pgtable, iova);
+
+ if (lv1ent_section(ent)) {
+ BUG_ON(size < SECT_SIZE);
+
+ *ent = 0;
+ pgtable_flush(ent, ent + 1);
+ size = SECT_SIZE;
+ goto done;
+ }
+
+ if (unlikely(lv1ent_fault(ent))) {
+ if (size > SECT_SIZE)
+ size = SECT_SIZE;
+ goto done;
+ }
+
+ /* lv1ent_page(sent) == true here */
+
+ ent = page_entry(ent, iova);
+
+ if (unlikely(lv2ent_fault(ent))) {
+ size = SPAGE_SIZE;
+ goto done;
+ }
+
+ if (lv2ent_small(ent)) {
+ *ent = 0;
+ size = SPAGE_SIZE;
+ priv->lv2entcnt[lv1ent_offset(iova)] += 1;
+ goto done;
+ }
+
+ /* lv1ent_large(ent) == true here */
+ BUG_ON(size < LPAGE_SIZE);
+
+ memset(ent, 0, sizeof(*ent) * SPAGES_PER_LPAGE);
+
+ size = LPAGE_SIZE;
+ priv->lv2entcnt[lv1ent_offset(iova)] += SPAGES_PER_LPAGE;
+done:
+ spin_unlock_irqrestore(&priv->pgtablelock, flags);
+
+ spin_lock_irqsave(&priv->lock, flags);
+ list_for_each_entry(data, &priv->clients, node)
+ sysmmu_tlb_invalidate_entry(data->dev, iova);
+ spin_unlock_irqrestore(&priv->lock, flags);
+
+
+ return size;
+}
+
+static phys_addr_t exynos_iommu_iova_to_phys(struct iommu_domain *domain,
+ unsigned long iova)
+{
+ struct exynos_iommu_domain *priv = domain->priv;
+ unsigned long *entry;
+ unsigned long flags;
+ phys_addr_t phys = 0;
+
+ spin_lock_irqsave(&priv->pgtablelock, flags);
+
+ entry = section_entry(priv->pgtable, iova);
+
+ if (lv1ent_section(entry)) {
+ phys = section_phys(entry) + section_offs(iova);
+ } else if (lv1ent_page(entry)) {
+ entry = page_entry(entry, iova);
+
+ if (lv2ent_large(entry))
+ phys = lpage_phys(entry) + lpage_offs(iova);
+ else if (lv2ent_small(entry))
+ phys = spage_phys(entry) + spage_offs(iova);
+ }
+
+ spin_unlock_irqrestore(&priv->pgtablelock, flags);
+
+ return phys;
+}
+
+static struct iommu_ops exynos_iommu_ops = {
+ .domain_init = &exynos_iommu_domain_init,
+ .domain_destroy = &exynos_iommu_domain_destroy,
+ .attach_dev = &exynos_iommu_attach_device,
+ .detach_dev = &exynos_iommu_detach_device,
+ .map = &exynos_iommu_map,
+ .unmap = &exynos_iommu_unmap,
+ .iova_to_phys = &exynos_iommu_iova_to_phys,
+ .pgsize_bitmap = SECT_SIZE | LPAGE_SIZE | SPAGE_SIZE,
+};
+
+static int __init exynos_iommu_init(void)
+{
+ int ret;
+
+ ret = platform_driver_register(&exynos_sysmmu_driver);
+
+ if (ret == 0)
+ bus_set_iommu(&platform_bus_type, &exynos_iommu_ops);
+
+ return ret;
+}
+arch_initcall(exynos_iommu_init);
depends on FRAMEBUFFER_CONSOLE || STI_CONSOLE
select FONT_8x16
select VIDEOBUF2_VMALLOC
+ select DMA_SHARED_BUFFER
default n
---help---
Enables a virtual video driver. This device shows a color bar
return vb2_qbuf(&fimc->vid_cap.vbq, buf);
}
+static int fimc_cap_expbuf(struct file *file, void *priv,
+ struct v4l2_exportbuffer *eb)
+{
+ struct fimc_dev *fimc = video_drvdata(file);
+
+ return vb2_expbuf(&fimc->vid_cap.vbq, eb);
+}
+
static int fimc_cap_dqbuf(struct file *file, void *priv,
struct v4l2_buffer *buf)
{
.vidioc_qbuf = fimc_cap_qbuf,
.vidioc_dqbuf = fimc_cap_dqbuf,
+ .vidioc_expbuf = fimc_cap_expbuf,
.vidioc_prepare_buf = fimc_cap_prepare_buf,
.vidioc_create_bufs = fimc_cap_create_bufs,
return -EINVAL;
}
+/* Export DMA buffer */
+static int vidioc_expbuf(struct file *file, void *priv,
+ struct v4l2_exportbuffer *eb)
+{
+ struct s5p_mfc_ctx *ctx = fh_to_ctx(priv);
+ int ret;
+
+ if (eb->mem_offset < DST_QUEUE_OFF_BASE)
+ return vb2_expbuf(&ctx->vq_src, eb);
+
+ eb->mem_offset -= DST_QUEUE_OFF_BASE;
+ ret = vb2_expbuf(&ctx->vq_dst, eb);
+ eb->mem_offset += DST_QUEUE_OFF_BASE;
+
+ return ret;
+}
+
/* Stream on */
static int vidioc_streamon(struct file *file, void *priv,
enum v4l2_buf_type type)
.vidioc_querybuf = vidioc_querybuf,
.vidioc_qbuf = vidioc_qbuf,
.vidioc_dqbuf = vidioc_dqbuf,
+ .vidioc_expbuf = vidioc_expbuf,
.vidioc_streamon = vidioc_streamon,
.vidioc_streamoff = vidioc_streamoff,
.vidioc_g_crop = vidioc_g_crop,
return -EINVAL;
}
+/* Export DMA buffer */
+static int vidioc_expbuf(struct file *file, void *priv,
+ struct v4l2_exportbuffer *eb)
+{
+ struct s5p_mfc_ctx *ctx = fh_to_ctx(priv);
+ int ret;
+
+ if (eb->mem_offset < DST_QUEUE_OFF_BASE)
+ return vb2_expbuf(&ctx->vq_src, eb);
+
+ eb->mem_offset -= DST_QUEUE_OFF_BASE;
+ ret = vb2_expbuf(&ctx->vq_dst, eb);
+ eb->mem_offset += DST_QUEUE_OFF_BASE;
+
+ return ret;
+}
+
/* Stream on */
static int vidioc_streamon(struct file *file, void *priv,
enum v4l2_buf_type type)
.vidioc_querybuf = vidioc_querybuf,
.vidioc_qbuf = vidioc_qbuf,
.vidioc_dqbuf = vidioc_dqbuf,
+ .vidioc_expbuf = vidioc_expbuf,
.vidioc_streamon = vidioc_streamon,
.vidioc_streamoff = vidioc_streamoff,
.vidioc_s_parm = vidioc_s_parm,
bool "Samsung TV driver for S5P platform (experimental)"
depends on PLAT_S5P && PM_RUNTIME
depends on EXPERIMENTAL
+ select DMA_SHARED_BUFFER
default n
---help---
Say Y here to enable selecting the TV output devices for
return vb2_dqbuf(&layer->vb_queue, p, file->f_flags & O_NONBLOCK);
}
+static int mxr_expbuf(struct file *file, void *priv,
+ struct v4l2_exportbuffer *eb)
+{
+ struct mxr_layer *layer = video_drvdata(file);
+
+ mxr_dbg(layer->mdev, "%s:%d\n", __func__, __LINE__);
+ return vb2_expbuf(&layer->vb_queue, eb);
+}
+
static int mxr_streamon(struct file *file, void *priv, enum v4l2_buf_type i)
{
struct mxr_layer *layer = video_drvdata(file);
.vidioc_querybuf = mxr_querybuf,
.vidioc_qbuf = mxr_qbuf,
.vidioc_dqbuf = mxr_dqbuf,
+ .vidioc_expbuf = mxr_expbuf,
/* Streaming control */
.vidioc_streamon = mxr_streamon,
.vidioc_streamoff = mxr_streamoff,
layer->vb_queue = (struct vb2_queue) {
.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE,
- .io_modes = VB2_MMAP | VB2_USERPTR,
+ .io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF,
.drv_priv = layer,
.buf_struct_size = sizeof(struct mxr_buffer),
.ops = &mxr_video_qops,
up_pln = compat_ptr(p);
if (put_user((unsigned long)up_pln, &up->m.userptr))
return -EFAULT;
+ } else if (memory == V4L2_MEMORY_DMABUF) {
+ if (copy_in_user(&up->m.fd, &up32->m.fd, sizeof(int)))
+ return -EFAULT;
} else {
if (copy_in_user(&up->m.mem_offset, &up32->m.mem_offset,
sizeof(__u32)))
if (copy_in_user(&up32->m.mem_offset, &up->m.mem_offset,
sizeof(__u32)))
return -EFAULT;
+ /* For DMABUF, driver might've set up the fd, so copy it back. */
+ if (memory == V4L2_MEMORY_DMABUF)
+ if (copy_in_user(&up32->m.fd, &up->m.fd,
+ sizeof(int)))
+ return -EFAULT;
return 0;
}
if (get_user(kp->m.offset, &up->m.offset))
return -EFAULT;
break;
+ case V4L2_MEMORY_DMABUF:
+ if (get_user(kp->m.fd, &up->m.fd))
+ return -EFAULT;
+ break;
}
}
if (put_user(kp->m.offset, &up->m.offset))
return -EFAULT;
break;
+ case V4L2_MEMORY_DMABUF:
+ if (put_user(kp->m.fd, &up->m.fd))
+ return -EFAULT;
+ break;
}
}
case VIDIOC_S_FBUF32:
case VIDIOC_OVERLAY32:
case VIDIOC_QBUF32:
+ case VIDIOC_EXPBUF:
case VIDIOC_DQBUF32:
case VIDIOC_STREAMON32:
case VIDIOC_STREAMOFF32:
int ret = -ENODEV;
if (vdev->fops->unlocked_ioctl) {
- if (vdev->lock && mutex_lock_interruptible(vdev->lock))
- return -ERESTARTSYS;
+ bool locked = false;
+
+ if (vdev->lock) {
+ /* always lock unless the cmd is marked as "don't use lock" */
+ locked = !v4l2_is_known_ioctl(cmd) ||
+ !test_bit(_IOC_NR(cmd), vdev->dont_use_lock);
+
+ if (locked && mutex_lock_interruptible(vdev->lock))
+ return -ERESTARTSYS;
+ }
if (video_is_registered(vdev))
ret = vdev->fops->unlocked_ioctl(filp, cmd, arg);
- if (vdev->lock)
+ if (locked)
mutex_unlock(vdev->lock);
} else if (vdev->fops->ioctl) {
/* This code path is a replacement for the BKL. It is a major
return find_first_zero_bit(used, VIDEO_NUM_DEVICES);
}
+#define SET_VALID_IOCTL(ops, cmd, op) \
+ if (ops->op) \
+ set_bit(_IOC_NR(cmd), valid_ioctls)
+
+/* This determines which ioctls are actually implemented in the driver.
+ It's a one-time thing which simplifies video_ioctl2 as it can just do
+ a bit test.
+
+ Note that drivers can override this by setting bits to 1 in
+ vdev->valid_ioctls. If an ioctl is marked as 1 when this function is
+ called, then that ioctl will actually be marked as unimplemented.
+
+ It does that by first setting up the local valid_ioctls bitmap, and
+ at the end do a:
+
+ vdev->valid_ioctls = valid_ioctls & ~(vdev->valid_ioctls)
+ */
+static void determine_valid_ioctls(struct video_device *vdev)
+{
+ DECLARE_BITMAP(valid_ioctls, BASE_VIDIOC_PRIVATE);
+ const struct v4l2_ioctl_ops *ops = vdev->ioctl_ops;
+
+ bitmap_zero(valid_ioctls, BASE_VIDIOC_PRIVATE);
+
+ SET_VALID_IOCTL(ops, VIDIOC_QUERYCAP, vidioc_querycap);
+ if (ops->vidioc_g_priority ||
+ test_bit(V4L2_FL_USE_FH_PRIO, &vdev->flags))
+ set_bit(_IOC_NR(VIDIOC_G_PRIORITY), valid_ioctls);
+ if (ops->vidioc_s_priority ||
+ test_bit(V4L2_FL_USE_FH_PRIO, &vdev->flags))
+ set_bit(_IOC_NR(VIDIOC_S_PRIORITY), valid_ioctls);
+ if (ops->vidioc_enum_fmt_vid_cap ||
+ ops->vidioc_enum_fmt_vid_out ||
+ ops->vidioc_enum_fmt_vid_cap_mplane ||
+ ops->vidioc_enum_fmt_vid_out_mplane ||
+ ops->vidioc_enum_fmt_vid_overlay ||
+ ops->vidioc_enum_fmt_type_private)
+ set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
+ if (ops->vidioc_g_fmt_vid_cap ||
+ ops->vidioc_g_fmt_vid_out ||
+ ops->vidioc_g_fmt_vid_cap_mplane ||
+ ops->vidioc_g_fmt_vid_out_mplane ||
+ ops->vidioc_g_fmt_vid_overlay ||
+ ops->vidioc_g_fmt_vbi_cap ||
+ ops->vidioc_g_fmt_vid_out_overlay ||
+ ops->vidioc_g_fmt_vbi_out ||
+ ops->vidioc_g_fmt_sliced_vbi_cap ||
+ ops->vidioc_g_fmt_sliced_vbi_out ||
+ ops->vidioc_g_fmt_type_private)
+ set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
+ if (ops->vidioc_s_fmt_vid_cap ||
+ ops->vidioc_s_fmt_vid_out ||
+ ops->vidioc_s_fmt_vid_cap_mplane ||
+ ops->vidioc_s_fmt_vid_out_mplane ||
+ ops->vidioc_s_fmt_vid_overlay ||
+ ops->vidioc_s_fmt_vbi_cap ||
+ ops->vidioc_s_fmt_vid_out_overlay ||
+ ops->vidioc_s_fmt_vbi_out ||
+ ops->vidioc_s_fmt_sliced_vbi_cap ||
+ ops->vidioc_s_fmt_sliced_vbi_out ||
+ ops->vidioc_s_fmt_type_private)
+ set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
+ if (ops->vidioc_try_fmt_vid_cap ||
+ ops->vidioc_try_fmt_vid_out ||
+ ops->vidioc_try_fmt_vid_cap_mplane ||
+ ops->vidioc_try_fmt_vid_out_mplane ||
+ ops->vidioc_try_fmt_vid_overlay ||
+ ops->vidioc_try_fmt_vbi_cap ||
+ ops->vidioc_try_fmt_vid_out_overlay ||
+ ops->vidioc_try_fmt_vbi_out ||
+ ops->vidioc_try_fmt_sliced_vbi_cap ||
+ ops->vidioc_try_fmt_sliced_vbi_out ||
+ ops->vidioc_try_fmt_type_private)
+ set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
+ SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
+ SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
+ SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
+ SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
+ SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
+ SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
+ SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
+ SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
+ SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
+ SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
+ if (vdev->tvnorms)
+ set_bit(_IOC_NR(VIDIOC_ENUMSTD), valid_ioctls);
+ if (ops->vidioc_g_std || vdev->current_norm)
+ set_bit(_IOC_NR(VIDIOC_G_STD), valid_ioctls);
+ SET_VALID_IOCTL(ops, VIDIOC_S_STD, vidioc_s_std);
+ SET_VALID_IOCTL(ops, VIDIOC_QUERYSTD, vidioc_querystd);
+ SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
+ SET_VALID_IOCTL(ops, VIDIOC_G_INPUT, vidioc_g_input);
+ SET_VALID_IOCTL(ops, VIDIOC_S_INPUT, vidioc_s_input);
+ SET_VALID_IOCTL(ops, VIDIOC_ENUMOUTPUT, vidioc_enum_output);
+ SET_VALID_IOCTL(ops, VIDIOC_G_OUTPUT, vidioc_g_output);
+ SET_VALID_IOCTL(ops, VIDIOC_S_OUTPUT, vidioc_s_output);
+ /* Note: the control handler can also be passed through the filehandle,
+ and that can't be tested here. If the bit for these control ioctls
+ is set, then the ioctl is valid. But if it is 0, then it can still
+ be valid if the filehandle passed the control handler. */
+ if (vdev->ctrl_handler || ops->vidioc_queryctrl)
+ set_bit(_IOC_NR(VIDIOC_QUERYCTRL), valid_ioctls);
+ if (vdev->ctrl_handler || ops->vidioc_g_ctrl || ops->vidioc_g_ext_ctrls)
+ set_bit(_IOC_NR(VIDIOC_G_CTRL), valid_ioctls);
+ if (vdev->ctrl_handler || ops->vidioc_s_ctrl || ops->vidioc_s_ext_ctrls)
+ set_bit(_IOC_NR(VIDIOC_S_CTRL), valid_ioctls);
+ if (vdev->ctrl_handler || ops->vidioc_g_ext_ctrls)
+ set_bit(_IOC_NR(VIDIOC_G_EXT_CTRLS), valid_ioctls);
+ if (vdev->ctrl_handler || ops->vidioc_s_ext_ctrls)
+ set_bit(_IOC_NR(VIDIOC_S_EXT_CTRLS), valid_ioctls);
+ if (vdev->ctrl_handler || ops->vidioc_try_ext_ctrls)
+ set_bit(_IOC_NR(VIDIOC_TRY_EXT_CTRLS), valid_ioctls);
+ if (vdev->ctrl_handler || ops->vidioc_querymenu)
+ set_bit(_IOC_NR(VIDIOC_QUERYMENU), valid_ioctls);
+ SET_VALID_IOCTL(ops, VIDIOC_ENUMAUDIO, vidioc_enumaudio);
+ SET_VALID_IOCTL(ops, VIDIOC_G_AUDIO, vidioc_g_audio);
+ SET_VALID_IOCTL(ops, VIDIOC_S_AUDIO, vidioc_s_audio);
+ SET_VALID_IOCTL(ops, VIDIOC_ENUMAUDOUT, vidioc_enumaudout);
+ SET_VALID_IOCTL(ops, VIDIOC_G_AUDOUT, vidioc_g_audout);
+ SET_VALID_IOCTL(ops, VIDIOC_S_AUDOUT, vidioc_s_audout);
+ SET_VALID_IOCTL(ops, VIDIOC_G_MODULATOR, vidioc_g_modulator);
+ SET_VALID_IOCTL(ops, VIDIOC_S_MODULATOR, vidioc_s_modulator);
+ if (ops->vidioc_g_crop || ops->vidioc_g_selection)
+ set_bit(_IOC_NR(VIDIOC_G_CROP), valid_ioctls);
+ if (ops->vidioc_s_crop || ops->vidioc_s_selection)
+ set_bit(_IOC_NR(VIDIOC_S_CROP), valid_ioctls);
+ SET_VALID_IOCTL(ops, VIDIOC_G_SELECTION, vidioc_g_selection);
+ SET_VALID_IOCTL(ops, VIDIOC_S_SELECTION, vidioc_s_selection);
+ if (ops->vidioc_cropcap || ops->vidioc_g_selection)
+ set_bit(_IOC_NR(VIDIOC_CROPCAP), valid_ioctls);
+ SET_VALID_IOCTL(ops, VIDIOC_G_JPEGCOMP, vidioc_g_jpegcomp);
+ SET_VALID_IOCTL(ops, VIDIOC_S_JPEGCOMP, vidioc_s_jpegcomp);
+ SET_VALID_IOCTL(ops, VIDIOC_G_ENC_INDEX, vidioc_g_enc_index);
+ SET_VALID_IOCTL(ops, VIDIOC_ENCODER_CMD, vidioc_encoder_cmd);
+ SET_VALID_IOCTL(ops, VIDIOC_TRY_ENCODER_CMD, vidioc_try_encoder_cmd);
+ SET_VALID_IOCTL(ops, VIDIOC_DECODER_CMD, vidioc_decoder_cmd);
+ SET_VALID_IOCTL(ops, VIDIOC_TRY_DECODER_CMD, vidioc_try_decoder_cmd);
+ if (ops->vidioc_g_parm || vdev->current_norm)
+ set_bit(_IOC_NR(VIDIOC_G_PARM), valid_ioctls);
+ SET_VALID_IOCTL(ops, VIDIOC_S_PARM, vidioc_s_parm);
+ SET_VALID_IOCTL(ops, VIDIOC_G_TUNER, vidioc_g_tuner);
+ SET_VALID_IOCTL(ops, VIDIOC_S_TUNER, vidioc_s_tuner);
+ SET_VALID_IOCTL(ops, VIDIOC_G_FREQUENCY, vidioc_g_frequency);
+ SET_VALID_IOCTL(ops, VIDIOC_S_FREQUENCY, vidioc_s_frequency);
+ SET_VALID_IOCTL(ops, VIDIOC_G_SLICED_VBI_CAP, vidioc_g_sliced_vbi_cap);
+ SET_VALID_IOCTL(ops, VIDIOC_LOG_STATUS, vidioc_log_status);
+#ifdef CONFIG_VIDEO_ADV_DEBUG
+ SET_VALID_IOCTL(ops, VIDIOC_DBG_G_REGISTER, vidioc_g_register);
+ SET_VALID_IOCTL(ops, VIDIOC_DBG_S_REGISTER, vidioc_s_register);
+#endif
+ SET_VALID_IOCTL(ops, VIDIOC_DBG_G_CHIP_IDENT, vidioc_g_chip_ident);
+ SET_VALID_IOCTL(ops, VIDIOC_S_HW_FREQ_SEEK, vidioc_s_hw_freq_seek);
+ SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
+ SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
+ SET_VALID_IOCTL(ops, VIDIOC_ENUM_DV_PRESETS, vidioc_enum_dv_presets);
+ SET_VALID_IOCTL(ops, VIDIOC_S_DV_PRESET, vidioc_s_dv_preset);
+ SET_VALID_IOCTL(ops, VIDIOC_G_DV_PRESET, vidioc_g_dv_preset);
+ SET_VALID_IOCTL(ops, VIDIOC_QUERY_DV_PRESET, vidioc_query_dv_preset);
+ SET_VALID_IOCTL(ops, VIDIOC_S_DV_TIMINGS, vidioc_s_dv_timings);
+ SET_VALID_IOCTL(ops, VIDIOC_G_DV_TIMINGS, vidioc_g_dv_timings);
+ /* yes, really vidioc_subscribe_event */
+ SET_VALID_IOCTL(ops, VIDIOC_DQEVENT, vidioc_subscribe_event);
+ SET_VALID_IOCTL(ops, VIDIOC_SUBSCRIBE_EVENT, vidioc_subscribe_event);
+ SET_VALID_IOCTL(ops, VIDIOC_UNSUBSCRIBE_EVENT, vidioc_unsubscribe_event);
+ SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
+ SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
+ bitmap_andnot(vdev->valid_ioctls, valid_ioctls, vdev->valid_ioctls,
+ BASE_VIDIOC_PRIVATE);
+}
+
/**
* __video_register_device - register video4linux devices
* @vdev: video device structure we want to register
vdev->index = get_index(vdev);
mutex_unlock(&videodev_lock);
+ if (vdev->ioctl_ops)
+ determine_valid_ioctls(vdev);
+
/* Part 3: Initialize the character device */
vdev->cdev = cdev_alloc();
if (vdev->cdev == NULL) {
memset((u8 *)(p) + offsetof(typeof(*(p)), field) + sizeof((p)->field), \
0, sizeof(*(p)) - offsetof(typeof(*(p)), field) - sizeof((p)->field))
-#define have_fmt_ops(foo) ( \
- ops->vidioc_##foo##_fmt_vid_cap || \
- ops->vidioc_##foo##_fmt_vid_out || \
- ops->vidioc_##foo##_fmt_vid_cap_mplane || \
- ops->vidioc_##foo##_fmt_vid_out_mplane || \
- ops->vidioc_##foo##_fmt_vid_overlay || \
- ops->vidioc_##foo##_fmt_vbi_cap || \
- ops->vidioc_##foo##_fmt_vid_out_overlay || \
- ops->vidioc_##foo##_fmt_vbi_out || \
- ops->vidioc_##foo##_fmt_sliced_vbi_cap || \
- ops->vidioc_##foo##_fmt_sliced_vbi_out || \
- ops->vidioc_##foo##_fmt_type_private)
-
struct std_descr {
v4l2_std_id std;
const char *descr;
[V4L2_MEMORY_MMAP] = "mmap",
[V4L2_MEMORY_USERPTR] = "userptr",
[V4L2_MEMORY_OVERLAY] = "overlay",
+ [V4L2_MEMORY_DMABUF] = "dmabuf",
};
#define prt_names(a, arr) ((((a) >= 0) && ((a) < ARRAY_SIZE(arr))) ? \
/* ------------------------------------------------------------------ */
/* debug help functions */
-static const char *v4l2_ioctls[] = {
- [_IOC_NR(VIDIOC_QUERYCAP)] = "VIDIOC_QUERYCAP",
- [_IOC_NR(VIDIOC_RESERVED)] = "VIDIOC_RESERVED",
- [_IOC_NR(VIDIOC_ENUM_FMT)] = "VIDIOC_ENUM_FMT",
- [_IOC_NR(VIDIOC_G_FMT)] = "VIDIOC_G_FMT",
- [_IOC_NR(VIDIOC_S_FMT)] = "VIDIOC_S_FMT",
- [_IOC_NR(VIDIOC_REQBUFS)] = "VIDIOC_REQBUFS",
- [_IOC_NR(VIDIOC_QUERYBUF)] = "VIDIOC_QUERYBUF",
- [_IOC_NR(VIDIOC_G_FBUF)] = "VIDIOC_G_FBUF",
- [_IOC_NR(VIDIOC_S_FBUF)] = "VIDIOC_S_FBUF",
- [_IOC_NR(VIDIOC_OVERLAY)] = "VIDIOC_OVERLAY",
- [_IOC_NR(VIDIOC_QBUF)] = "VIDIOC_QBUF",
- [_IOC_NR(VIDIOC_DQBUF)] = "VIDIOC_DQBUF",
- [_IOC_NR(VIDIOC_STREAMON)] = "VIDIOC_STREAMON",
- [_IOC_NR(VIDIOC_STREAMOFF)] = "VIDIOC_STREAMOFF",
- [_IOC_NR(VIDIOC_G_PARM)] = "VIDIOC_G_PARM",
- [_IOC_NR(VIDIOC_S_PARM)] = "VIDIOC_S_PARM",
- [_IOC_NR(VIDIOC_G_STD)] = "VIDIOC_G_STD",
- [_IOC_NR(VIDIOC_S_STD)] = "VIDIOC_S_STD",
- [_IOC_NR(VIDIOC_ENUMSTD)] = "VIDIOC_ENUMSTD",
- [_IOC_NR(VIDIOC_ENUMINPUT)] = "VIDIOC_ENUMINPUT",
- [_IOC_NR(VIDIOC_G_CTRL)] = "VIDIOC_G_CTRL",
- [_IOC_NR(VIDIOC_S_CTRL)] = "VIDIOC_S_CTRL",
- [_IOC_NR(VIDIOC_G_TUNER)] = "VIDIOC_G_TUNER",
- [_IOC_NR(VIDIOC_S_TUNER)] = "VIDIOC_S_TUNER",
- [_IOC_NR(VIDIOC_G_AUDIO)] = "VIDIOC_G_AUDIO",
- [_IOC_NR(VIDIOC_S_AUDIO)] = "VIDIOC_S_AUDIO",
- [_IOC_NR(VIDIOC_QUERYCTRL)] = "VIDIOC_QUERYCTRL",
- [_IOC_NR(VIDIOC_QUERYMENU)] = "VIDIOC_QUERYMENU",
- [_IOC_NR(VIDIOC_G_INPUT)] = "VIDIOC_G_INPUT",
- [_IOC_NR(VIDIOC_S_INPUT)] = "VIDIOC_S_INPUT",
- [_IOC_NR(VIDIOC_G_OUTPUT)] = "VIDIOC_G_OUTPUT",
- [_IOC_NR(VIDIOC_S_OUTPUT)] = "VIDIOC_S_OUTPUT",
- [_IOC_NR(VIDIOC_ENUMOUTPUT)] = "VIDIOC_ENUMOUTPUT",
- [_IOC_NR(VIDIOC_G_AUDOUT)] = "VIDIOC_G_AUDOUT",
- [_IOC_NR(VIDIOC_S_AUDOUT)] = "VIDIOC_S_AUDOUT",
- [_IOC_NR(VIDIOC_G_MODULATOR)] = "VIDIOC_G_MODULATOR",
- [_IOC_NR(VIDIOC_S_MODULATOR)] = "VIDIOC_S_MODULATOR",
- [_IOC_NR(VIDIOC_G_FREQUENCY)] = "VIDIOC_G_FREQUENCY",
- [_IOC_NR(VIDIOC_S_FREQUENCY)] = "VIDIOC_S_FREQUENCY",
- [_IOC_NR(VIDIOC_CROPCAP)] = "VIDIOC_CROPCAP",
- [_IOC_NR(VIDIOC_G_CROP)] = "VIDIOC_G_CROP",
- [_IOC_NR(VIDIOC_S_CROP)] = "VIDIOC_S_CROP",
- [_IOC_NR(VIDIOC_G_SELECTION)] = "VIDIOC_G_SELECTION",
- [_IOC_NR(VIDIOC_S_SELECTION)] = "VIDIOC_S_SELECTION",
- [_IOC_NR(VIDIOC_G_JPEGCOMP)] = "VIDIOC_G_JPEGCOMP",
- [_IOC_NR(VIDIOC_S_JPEGCOMP)] = "VIDIOC_S_JPEGCOMP",
- [_IOC_NR(VIDIOC_QUERYSTD)] = "VIDIOC_QUERYSTD",
- [_IOC_NR(VIDIOC_TRY_FMT)] = "VIDIOC_TRY_FMT",
- [_IOC_NR(VIDIOC_ENUMAUDIO)] = "VIDIOC_ENUMAUDIO",
- [_IOC_NR(VIDIOC_ENUMAUDOUT)] = "VIDIOC_ENUMAUDOUT",
- [_IOC_NR(VIDIOC_G_PRIORITY)] = "VIDIOC_G_PRIORITY",
- [_IOC_NR(VIDIOC_S_PRIORITY)] = "VIDIOC_S_PRIORITY",
- [_IOC_NR(VIDIOC_G_SLICED_VBI_CAP)] = "VIDIOC_G_SLICED_VBI_CAP",
- [_IOC_NR(VIDIOC_LOG_STATUS)] = "VIDIOC_LOG_STATUS",
- [_IOC_NR(VIDIOC_G_EXT_CTRLS)] = "VIDIOC_G_EXT_CTRLS",
- [_IOC_NR(VIDIOC_S_EXT_CTRLS)] = "VIDIOC_S_EXT_CTRLS",
- [_IOC_NR(VIDIOC_TRY_EXT_CTRLS)] = "VIDIOC_TRY_EXT_CTRLS",
-#if 1
- [_IOC_NR(VIDIOC_ENUM_FRAMESIZES)] = "VIDIOC_ENUM_FRAMESIZES",
- [_IOC_NR(VIDIOC_ENUM_FRAMEINTERVALS)] = "VIDIOC_ENUM_FRAMEINTERVALS",
- [_IOC_NR(VIDIOC_G_ENC_INDEX)] = "VIDIOC_G_ENC_INDEX",
- [_IOC_NR(VIDIOC_ENCODER_CMD)] = "VIDIOC_ENCODER_CMD",
- [_IOC_NR(VIDIOC_TRY_ENCODER_CMD)] = "VIDIOC_TRY_ENCODER_CMD",
-
- [_IOC_NR(VIDIOC_DECODER_CMD)] = "VIDIOC_DECODER_CMD",
- [_IOC_NR(VIDIOC_TRY_DECODER_CMD)] = "VIDIOC_TRY_DECODER_CMD",
- [_IOC_NR(VIDIOC_DBG_S_REGISTER)] = "VIDIOC_DBG_S_REGISTER",
- [_IOC_NR(VIDIOC_DBG_G_REGISTER)] = "VIDIOC_DBG_G_REGISTER",
-
- [_IOC_NR(VIDIOC_DBG_G_CHIP_IDENT)] = "VIDIOC_DBG_G_CHIP_IDENT",
- [_IOC_NR(VIDIOC_S_HW_FREQ_SEEK)] = "VIDIOC_S_HW_FREQ_SEEK",
+
+struct v4l2_ioctl_info {
+ unsigned int ioctl;
+ u16 flags;
+ const char * const name;
+};
+
+/* This control needs a priority check */
+#define INFO_FL_PRIO (1 << 0)
+/* This control can be valid if the filehandle passes a control handler. */
+#define INFO_FL_CTRL (1 << 1)
+
+#define IOCTL_INFO(_ioctl, _flags) [_IOC_NR(_ioctl)] = { \
+ .ioctl = _ioctl, \
+ .flags = _flags, \
+ .name = #_ioctl, \
+}
+
+static struct v4l2_ioctl_info v4l2_ioctls[] = {
+ IOCTL_INFO(VIDIOC_QUERYCAP, 0),
+ IOCTL_INFO(VIDIOC_ENUM_FMT, 0),
+ IOCTL_INFO(VIDIOC_G_FMT, 0),
+ IOCTL_INFO(VIDIOC_S_FMT, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_REQBUFS, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_QUERYBUF, 0),
+ IOCTL_INFO(VIDIOC_G_FBUF, 0),
+ IOCTL_INFO(VIDIOC_S_FBUF, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_OVERLAY, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_QBUF, 0),
+ IOCTL_INFO(VIDIOC_EXPBUF, 0),
+ IOCTL_INFO(VIDIOC_DQBUF, 0),
+ IOCTL_INFO(VIDIOC_STREAMON, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_STREAMOFF, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_G_PARM, 0),
+ IOCTL_INFO(VIDIOC_S_PARM, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_G_STD, 0),
+ IOCTL_INFO(VIDIOC_S_STD, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_ENUMSTD, 0),
+ IOCTL_INFO(VIDIOC_ENUMINPUT, 0),
+ IOCTL_INFO(VIDIOC_G_CTRL, INFO_FL_CTRL),
+ IOCTL_INFO(VIDIOC_S_CTRL, INFO_FL_PRIO | INFO_FL_CTRL),
+ IOCTL_INFO(VIDIOC_G_TUNER, 0),
+ IOCTL_INFO(VIDIOC_S_TUNER, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_G_AUDIO, 0),
+ IOCTL_INFO(VIDIOC_S_AUDIO, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_QUERYCTRL, INFO_FL_CTRL),
+ IOCTL_INFO(VIDIOC_QUERYMENU, INFO_FL_CTRL),
+ IOCTL_INFO(VIDIOC_G_INPUT, 0),
+ IOCTL_INFO(VIDIOC_S_INPUT, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_G_OUTPUT, 0),
+ IOCTL_INFO(VIDIOC_S_OUTPUT, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_ENUMOUTPUT, 0),
+ IOCTL_INFO(VIDIOC_G_AUDOUT, 0),
+ IOCTL_INFO(VIDIOC_S_AUDOUT, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_G_MODULATOR, 0),
+ IOCTL_INFO(VIDIOC_S_MODULATOR, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_G_FREQUENCY, 0),
+ IOCTL_INFO(VIDIOC_S_FREQUENCY, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_CROPCAP, 0),
+ IOCTL_INFO(VIDIOC_G_CROP, 0),
+ IOCTL_INFO(VIDIOC_S_CROP, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_G_SELECTION, 0),
+ IOCTL_INFO(VIDIOC_S_SELECTION, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_G_JPEGCOMP, 0),
+ IOCTL_INFO(VIDIOC_S_JPEGCOMP, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_QUERYSTD, 0),
+ IOCTL_INFO(VIDIOC_TRY_FMT, 0),
+ IOCTL_INFO(VIDIOC_ENUMAUDIO, 0),
+ IOCTL_INFO(VIDIOC_ENUMAUDOUT, 0),
+ IOCTL_INFO(VIDIOC_G_PRIORITY, 0),
+ IOCTL_INFO(VIDIOC_S_PRIORITY, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_G_SLICED_VBI_CAP, 0),
+ IOCTL_INFO(VIDIOC_LOG_STATUS, 0),
+ IOCTL_INFO(VIDIOC_G_EXT_CTRLS, INFO_FL_CTRL),
+ IOCTL_INFO(VIDIOC_S_EXT_CTRLS, INFO_FL_PRIO | INFO_FL_CTRL),
+ IOCTL_INFO(VIDIOC_TRY_EXT_CTRLS, 0),
+ IOCTL_INFO(VIDIOC_ENUM_FRAMESIZES, 0),
+ IOCTL_INFO(VIDIOC_ENUM_FRAMEINTERVALS, 0),
+ IOCTL_INFO(VIDIOC_G_ENC_INDEX, 0),
+ IOCTL_INFO(VIDIOC_ENCODER_CMD, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_TRY_ENCODER_CMD, 0),
+ IOCTL_INFO(VIDIOC_DECODER_CMD, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_TRY_DECODER_CMD, 0),
+#ifdef CONFIG_VIDEO_ADV_DEBUG
+ IOCTL_INFO(VIDIOC_DBG_S_REGISTER, 0),
+ IOCTL_INFO(VIDIOC_DBG_G_REGISTER, 0),
#endif
- [_IOC_NR(VIDIOC_ENUM_DV_PRESETS)] = "VIDIOC_ENUM_DV_PRESETS",
- [_IOC_NR(VIDIOC_S_DV_PRESET)] = "VIDIOC_S_DV_PRESET",
- [_IOC_NR(VIDIOC_G_DV_PRESET)] = "VIDIOC_G_DV_PRESET",
- [_IOC_NR(VIDIOC_QUERY_DV_PRESET)] = "VIDIOC_QUERY_DV_PRESET",
- [_IOC_NR(VIDIOC_S_DV_TIMINGS)] = "VIDIOC_S_DV_TIMINGS",
- [_IOC_NR(VIDIOC_G_DV_TIMINGS)] = "VIDIOC_G_DV_TIMINGS",
- [_IOC_NR(VIDIOC_DQEVENT)] = "VIDIOC_DQEVENT",
- [_IOC_NR(VIDIOC_SUBSCRIBE_EVENT)] = "VIDIOC_SUBSCRIBE_EVENT",
- [_IOC_NR(VIDIOC_UNSUBSCRIBE_EVENT)] = "VIDIOC_UNSUBSCRIBE_EVENT",
- [_IOC_NR(VIDIOC_CREATE_BUFS)] = "VIDIOC_CREATE_BUFS",
- [_IOC_NR(VIDIOC_PREPARE_BUF)] = "VIDIOC_PREPARE_BUF",
+ IOCTL_INFO(VIDIOC_DBG_G_CHIP_IDENT, 0),
+ IOCTL_INFO(VIDIOC_S_HW_FREQ_SEEK, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_ENUM_DV_PRESETS, 0),
+ IOCTL_INFO(VIDIOC_S_DV_PRESET, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_G_DV_PRESET, 0),
+ IOCTL_INFO(VIDIOC_QUERY_DV_PRESET, 0),
+ IOCTL_INFO(VIDIOC_S_DV_TIMINGS, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_G_DV_TIMINGS, 0),
+ IOCTL_INFO(VIDIOC_DQEVENT, 0),
+ IOCTL_INFO(VIDIOC_SUBSCRIBE_EVENT, 0),
+ IOCTL_INFO(VIDIOC_UNSUBSCRIBE_EVENT, 0),
+ IOCTL_INFO(VIDIOC_CREATE_BUFS, INFO_FL_PRIO),
+ IOCTL_INFO(VIDIOC_PREPARE_BUF, 0),
};
#define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls)
+bool v4l2_is_known_ioctl(unsigned int cmd)
+{
+ if (_IOC_NR(cmd) >= V4L2_IOCTLS)
+ return false;
+ return v4l2_ioctls[_IOC_NR(cmd)].ioctl == cmd;
+}
+
/* Common ioctl debug function. This function can be used by
external ioctl messages as well as internal V4L ioctl */
void v4l_printk_ioctl(unsigned int cmd)
type = "v4l2";
break;
}
- printk("%s", v4l2_ioctls[_IOC_NR(cmd)]);
+ printk("%s", v4l2_ioctls[_IOC_NR(cmd)].name);
return;
default:
type = "unknown";
void *fh = file->private_data;
struct v4l2_fh *vfh = NULL;
int use_fh_prio = 0;
- long ret_prio = 0;
long ret = -ENOTTY;
if (ops == NULL) {
return ret;
}
- if ((vfd->debug & V4L2_DEBUG_IOCTL) &&
- !(vfd->debug & V4L2_DEBUG_IOCTL_ARG)) {
- v4l_print_ioctl(vfd->name, cmd);
- printk(KERN_CONT "\n");
- }
-
if (test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags)) {
vfh = file->private_data;
use_fh_prio = test_bit(V4L2_FL_USE_FH_PRIO, &vfd->flags);
}
- if (use_fh_prio)
- ret_prio = v4l2_prio_check(vfd->prio, vfh->prio);
+ if (v4l2_is_known_ioctl(cmd)) {
+ struct v4l2_ioctl_info *info = &v4l2_ioctls[_IOC_NR(cmd)];
+
+ if (!test_bit(_IOC_NR(cmd), vfd->valid_ioctls) &&
+ !((info->flags & INFO_FL_CTRL) && vfh && vfh->ctrl_handler))
+ return -ENOTTY;
+
+ if (use_fh_prio && (info->flags & INFO_FL_PRIO)) {
+ ret = v4l2_prio_check(vfd->prio, vfh->prio);
+ if (ret)
+ return ret;
+ }
+ }
+
+ if ((vfd->debug & V4L2_DEBUG_IOCTL) &&
+ !(vfd->debug & V4L2_DEBUG_IOCTL_ARG)) {
+ v4l_print_ioctl(vfd->name, cmd);
+ printk(KERN_CONT "\n");
+ }
switch (cmd) {
{
struct v4l2_capability *cap = (struct v4l2_capability *)arg;
- if (!ops->vidioc_querycap)
- break;
-
cap->version = LINUX_VERSION_CODE;
ret = ops->vidioc_querycap(file, fh, cap);
if (!ret)
{
enum v4l2_priority *p = arg;
- if (!ops->vidioc_s_priority && !use_fh_prio)
- break;
dbgarg(cmd, "setting priority to %d\n", *p);
if (ops->vidioc_s_priority)
ret = ops->vidioc_s_priority(file, fh, *p);
else
- ret = ret_prio ? ret_prio :
- v4l2_prio_change(&vfd->v4l2_dev->prio,
+ ret = v4l2_prio_change(&vfd->v4l2_dev->prio,
&vfh->prio, *p);
break;
}
{
struct v4l2_fmtdesc *f = arg;
+ ret = -EINVAL;
switch (f->type) {
case V4L2_BUF_TYPE_VIDEO_CAPTURE:
if (likely(ops->vidioc_enum_fmt_vid_cap))
default:
break;
}
- if (likely (!ret))
+ if (likely(!ret))
dbgarg(cmd, "index=%d, type=%d, flags=%d, "
"pixelformat=%c%c%c%c, description='%s'\n",
f->index, f->type, f->flags,
(f->pixelformat >> 16) & 0xff,
(f->pixelformat >> 24) & 0xff,
f->description);
- else if (ret == -ENOTTY &&
- (ops->vidioc_enum_fmt_vid_cap ||
- ops->vidioc_enum_fmt_vid_out ||
- ops->vidioc_enum_fmt_vid_cap_mplane ||
- ops->vidioc_enum_fmt_vid_out_mplane ||
- ops->vidioc_enum_fmt_vid_overlay ||
- ops->vidioc_enum_fmt_type_private))
- ret = -EINVAL;
break;
}
case VIDIOC_G_FMT:
/* FIXME: Should be one dump per type */
dbgarg(cmd, "type=%s\n", prt_names(f->type, v4l2_type_names));
+ ret = -EINVAL;
switch (f->type) {
case V4L2_BUF_TYPE_VIDEO_CAPTURE:
if (ops->vidioc_g_fmt_vid_cap)
fh, f);
break;
}
- if (unlikely(ret == -ENOTTY && have_fmt_ops(g)))
- ret = -EINVAL;
-
break;
}
case VIDIOC_S_FMT:
{
struct v4l2_format *f = (struct v4l2_format *)arg;
- if (!have_fmt_ops(s))
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
ret = -EINVAL;
/* FIXME: Should be one dump per type */
/* FIXME: Should be one dump per type */
dbgarg(cmd, "type=%s\n", prt_names(f->type,
v4l2_type_names));
+ ret = -EINVAL;
switch (f->type) {
case V4L2_BUF_TYPE_VIDEO_CAPTURE:
CLEAR_AFTER_FIELD(f, fmt.pix);
fh, f);
break;
}
- if (unlikely(ret == -ENOTTY && have_fmt_ops(try)))
- ret = -EINVAL;
break;
}
/* FIXME: Those buf reqs could be handled here,
{
struct v4l2_requestbuffers *p = arg;
- if (!ops->vidioc_reqbufs)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
ret = check_fmt(ops, p->type);
if (ret)
break;
{
struct v4l2_buffer *p = arg;
- if (!ops->vidioc_querybuf)
- break;
ret = check_fmt(ops, p->type);
if (ret)
break;
{
struct v4l2_buffer *p = arg;
- if (!ops->vidioc_qbuf)
- break;
ret = check_fmt(ops, p->type);
if (ret)
break;
dbgbuf(cmd, vfd, p);
break;
}
+ case VIDIOC_EXPBUF:
+ {
+ ret = ops->vidioc_expbuf(file, fh, arg);
+ break;
+ }
case VIDIOC_DQBUF:
{
struct v4l2_buffer *p = arg;
- if (!ops->vidioc_dqbuf)
- break;
ret = check_fmt(ops, p->type);
if (ret)
break;
{
int *i = arg;
- if (!ops->vidioc_overlay)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
dbgarg(cmd, "value=%d\n", *i);
ret = ops->vidioc_overlay(file, fh, *i);
break;
{
struct v4l2_framebuffer *p = arg;
- if (!ops->vidioc_g_fbuf)
- break;
ret = ops->vidioc_g_fbuf(file, fh, arg);
if (!ret) {
dbgarg(cmd, "capability=0x%x, flags=%d, base=0x%08lx\n",
{
struct v4l2_framebuffer *p = arg;
- if (!ops->vidioc_s_fbuf)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
dbgarg(cmd, "capability=0x%x, flags=%d, base=0x%08lx\n",
p->capability, p->flags, (unsigned long)p->base);
v4l_print_pix_fmt(vfd, &p->fmt);
{
enum v4l2_buf_type i = *(int *)arg;
- if (!ops->vidioc_streamon)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
dbgarg(cmd, "type=%s\n", prt_names(i, v4l2_type_names));
ret = ops->vidioc_streamon(file, fh, i);
break;
{
enum v4l2_buf_type i = *(int *)arg;
- if (!ops->vidioc_streamoff)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
dbgarg(cmd, "type=%s\n", prt_names(i, v4l2_type_names));
ret = ops->vidioc_streamoff(file, fh, i);
break;
dbgarg(cmd, "std=%08Lx\n", (long long unsigned)*id);
- if (!ops->vidioc_s_std)
- break;
-
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
ret = -EINVAL;
norm = (*id) & vfd->tvnorms;
if (vfd->tvnorms && !norm) /* Check if std is supported */
{
v4l2_std_id *p = arg;
- if (!ops->vidioc_querystd)
- break;
/*
* If nothing detected, it should return all supported
* Drivers just need to mask the std argument, in order
if (ops->vidioc_s_dv_timings)
p->capabilities |= V4L2_IN_CAP_CUSTOM_TIMINGS;
- if (!ops->vidioc_enum_input)
- break;
-
ret = ops->vidioc_enum_input(file, fh, p);
if (!ret)
dbgarg(cmd, "index=%d, name=%s, type=%d, "
{
unsigned int *i = arg;
- if (!ops->vidioc_g_input)
- break;
ret = ops->vidioc_g_input(file, fh, i);
if (!ret)
dbgarg(cmd, "value=%d\n", *i);
{
unsigned int *i = arg;
- if (!ops->vidioc_s_input)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
dbgarg(cmd, "value=%d\n", *i);
ret = ops->vidioc_s_input(file, fh, *i);
break;
{
struct v4l2_output *p = arg;
- if (!ops->vidioc_enum_output)
- break;
-
/*
* We set the flags for CAP_PRESETS, CAP_CUSTOM_TIMINGS &
* CAP_STD here based on ioctl handler provided by the
{
unsigned int *i = arg;
- if (!ops->vidioc_g_output)
- break;
ret = ops->vidioc_g_output(file, fh, i);
if (!ret)
dbgarg(cmd, "value=%d\n", *i);
{
unsigned int *i = arg;
- if (!ops->vidioc_s_output)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
dbgarg(cmd, "value=%d\n", *i);
ret = ops->vidioc_s_output(file, fh, *i);
break;
if (!(vfh && vfh->ctrl_handler) && !vfd->ctrl_handler &&
!ops->vidioc_s_ctrl && !ops->vidioc_s_ext_ctrls)
break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
dbgarg(cmd, "id=0x%x, value=%d\n", p->id, p->value);
if (!(vfh && vfh->ctrl_handler) && !vfd->ctrl_handler &&
!ops->vidioc_s_ext_ctrls)
break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
v4l_print_ext_ctrls(cmd, vfd, p, 1);
if (vfh && vfh->ctrl_handler)
ret = v4l2_s_ext_ctrls(vfh, vfh->ctrl_handler, p);
{
struct v4l2_audio *p = arg;
- if (!ops->vidioc_enumaudio)
- break;
ret = ops->vidioc_enumaudio(file, fh, p);
if (!ret)
dbgarg(cmd, "index=%d, name=%s, capability=0x%x, "
{
struct v4l2_audio *p = arg;
- if (!ops->vidioc_g_audio)
- break;
-
ret = ops->vidioc_g_audio(file, fh, p);
if (!ret)
dbgarg(cmd, "index=%d, name=%s, capability=0x%x, "
{
struct v4l2_audio *p = arg;
- if (!ops->vidioc_s_audio)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
dbgarg(cmd, "index=%d, name=%s, capability=0x%x, "
"mode=0x%x\n", p->index, p->name,
p->capability, p->mode);
{
struct v4l2_audioout *p = arg;
- if (!ops->vidioc_enumaudout)
- break;
dbgarg(cmd, "Enum for index=%d\n", p->index);
ret = ops->vidioc_enumaudout(file, fh, p);
if (!ret)
{
struct v4l2_audioout *p = arg;
- if (!ops->vidioc_g_audout)
- break;
-
ret = ops->vidioc_g_audout(file, fh, p);
if (!ret)
dbgarg2("index=%d, name=%s, capability=%d, "
{
struct v4l2_audioout *p = arg;
- if (!ops->vidioc_s_audout)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
dbgarg(cmd, "index=%d, name=%s, capability=%d, "
"mode=%d\n", p->index, p->name,
p->capability, p->mode);
{
struct v4l2_modulator *p = arg;
- if (!ops->vidioc_g_modulator)
- break;
ret = ops->vidioc_g_modulator(file, fh, p);
if (!ret)
dbgarg(cmd, "index=%d, name=%s, "
{
struct v4l2_modulator *p = arg;
- if (!ops->vidioc_s_modulator)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
dbgarg(cmd, "index=%d, name=%s, capability=%d, "
"rangelow=%d, rangehigh=%d, txsubchans=%d\n",
p->index, p->name, p->capability, p->rangelow,
{
struct v4l2_crop *p = arg;
- if (!ops->vidioc_g_crop && !ops->vidioc_g_selection)
- break;
-
dbgarg(cmd, "type=%s\n", prt_names(p->type, v4l2_type_names));
if (ops->vidioc_g_crop) {
{
struct v4l2_crop *p = arg;
- if (!ops->vidioc_s_crop && !ops->vidioc_s_selection)
- break;
-
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
dbgarg(cmd, "type=%s\n", prt_names(p->type, v4l2_type_names));
dbgrect(vfd, "", &p->c);
{
struct v4l2_selection *p = arg;
- if (!ops->vidioc_g_selection)
- break;
-
dbgarg(cmd, "type=%s\n", prt_names(p->type, v4l2_type_names));
ret = ops->vidioc_g_selection(file, fh, p);
{
struct v4l2_selection *p = arg;
- if (!ops->vidioc_s_selection)
- break;
-
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
dbgarg(cmd, "type=%s\n", prt_names(p->type, v4l2_type_names));
dbgrect(vfd, "", &p->r);
struct v4l2_cropcap *p = arg;
/*FIXME: Should also show v4l2_fract pixelaspect */
- if (!ops->vidioc_cropcap && !ops->vidioc_g_selection)
- break;
-
dbgarg(cmd, "type=%s\n", prt_names(p->type, v4l2_type_names));
if (ops->vidioc_cropcap) {
ret = ops->vidioc_cropcap(file, fh, p);
{
struct v4l2_jpegcompression *p = arg;
- if (!ops->vidioc_g_jpegcomp)
- break;
-
ret = ops->vidioc_g_jpegcomp(file, fh, p);
if (!ret)
dbgarg(cmd, "quality=%d, APPn=%d, "
{
struct v4l2_jpegcompression *p = arg;
- if (!ops->vidioc_g_jpegcomp)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
dbgarg(cmd, "quality=%d, APPn=%d, APP_len=%d, "
"COM_len=%d, jpeg_markers=%d\n",
p->quality, p->APPn, p->APP_len,
{
struct v4l2_enc_idx *p = arg;
- if (!ops->vidioc_g_enc_index)
- break;
ret = ops->vidioc_g_enc_index(file, fh, p);
if (!ret)
dbgarg(cmd, "entries=%d, entries_cap=%d\n",
{
struct v4l2_encoder_cmd *p = arg;
- if (!ops->vidioc_encoder_cmd)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
ret = ops->vidioc_encoder_cmd(file, fh, p);
if (!ret)
dbgarg(cmd, "cmd=%d, flags=%x\n", p->cmd, p->flags);
{
struct v4l2_encoder_cmd *p = arg;
- if (!ops->vidioc_try_encoder_cmd)
- break;
ret = ops->vidioc_try_encoder_cmd(file, fh, p);
if (!ret)
dbgarg(cmd, "cmd=%d, flags=%x\n", p->cmd, p->flags);
{
struct v4l2_decoder_cmd *p = arg;
- if (!ops->vidioc_decoder_cmd)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
ret = ops->vidioc_decoder_cmd(file, fh, p);
if (!ret)
dbgarg(cmd, "cmd=%d, flags=%x\n", p->cmd, p->flags);
{
struct v4l2_decoder_cmd *p = arg;
- if (!ops->vidioc_try_decoder_cmd)
- break;
ret = ops->vidioc_try_decoder_cmd(file, fh, p);
if (!ret)
dbgarg(cmd, "cmd=%d, flags=%x\n", p->cmd, p->flags);
{
struct v4l2_streamparm *p = arg;
- if (!ops->vidioc_g_parm && !vfd->current_norm)
- break;
if (ops->vidioc_g_parm) {
ret = check_fmt(ops, p->type);
if (ret)
{
struct v4l2_streamparm *p = arg;
- if (!ops->vidioc_s_parm)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
ret = check_fmt(ops, p->type);
if (ret)
break;
{
struct v4l2_tuner *p = arg;
- if (!ops->vidioc_g_tuner)
- break;
-
p->type = (vfd->vfl_type == VFL_TYPE_RADIO) ?
V4L2_TUNER_RADIO : V4L2_TUNER_ANALOG_TV;
ret = ops->vidioc_g_tuner(file, fh, p);
{
struct v4l2_tuner *p = arg;
- if (!ops->vidioc_s_tuner)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
p->type = (vfd->vfl_type == VFL_TYPE_RADIO) ?
V4L2_TUNER_RADIO : V4L2_TUNER_ANALOG_TV;
dbgarg(cmd, "index=%d, name=%s, type=%d, "
{
struct v4l2_frequency *p = arg;
- if (!ops->vidioc_g_frequency)
- break;
-
p->type = (vfd->vfl_type == VFL_TYPE_RADIO) ?
V4L2_TUNER_RADIO : V4L2_TUNER_ANALOG_TV;
ret = ops->vidioc_g_frequency(file, fh, p);
struct v4l2_frequency *p = arg;
enum v4l2_tuner_type type;
- if (!ops->vidioc_s_frequency)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
type = (vfd->vfl_type == VFL_TYPE_RADIO) ?
V4L2_TUNER_RADIO : V4L2_TUNER_ANALOG_TV;
dbgarg(cmd, "tuner=%d, type=%d, frequency=%d\n",
{
struct v4l2_sliced_vbi_cap *p = arg;
- if (!ops->vidioc_g_sliced_vbi_cap)
- break;
-
/* Clear up to type, everything after type is zerod already */
memset(p, 0, offsetof(struct v4l2_sliced_vbi_cap, type));
}
case VIDIOC_LOG_STATUS:
{
- if (!ops->vidioc_log_status)
- break;
if (vfd->v4l2_dev)
pr_info("%s: ================= START STATUS =================\n",
vfd->v4l2_dev->name);
vfd->v4l2_dev->name);
break;
}
-#ifdef CONFIG_VIDEO_ADV_DEBUG
case VIDIOC_DBG_G_REGISTER:
{
+#ifdef CONFIG_VIDEO_ADV_DEBUG
struct v4l2_dbg_register *p = arg;
- if (ops->vidioc_g_register) {
- if (!capable(CAP_SYS_ADMIN))
- ret = -EPERM;
- else
- ret = ops->vidioc_g_register(file, fh, p);
- }
+ if (!capable(CAP_SYS_ADMIN))
+ ret = -EPERM;
+ else
+ ret = ops->vidioc_g_register(file, fh, p);
+#endif
break;
}
case VIDIOC_DBG_S_REGISTER:
{
+#ifdef CONFIG_VIDEO_ADV_DEBUG
struct v4l2_dbg_register *p = arg;
- if (ops->vidioc_s_register) {
- if (!capable(CAP_SYS_ADMIN))
- ret = -EPERM;
- else
- ret = ops->vidioc_s_register(file, fh, p);
- }
+ if (!capable(CAP_SYS_ADMIN))
+ ret = -EPERM;
+ else
+ ret = ops->vidioc_s_register(file, fh, p);
+#endif
break;
}
-#endif
case VIDIOC_DBG_G_CHIP_IDENT:
{
struct v4l2_dbg_chip_ident *p = arg;
- if (!ops->vidioc_g_chip_ident)
- break;
p->ident = V4L2_IDENT_NONE;
p->revision = 0;
ret = ops->vidioc_g_chip_ident(file, fh, p);
struct v4l2_hw_freq_seek *p = arg;
enum v4l2_tuner_type type;
- if (!ops->vidioc_s_hw_freq_seek)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
type = (vfd->vfl_type == VFL_TYPE_RADIO) ?
V4L2_TUNER_RADIO : V4L2_TUNER_ANALOG_TV;
dbgarg(cmd,
{
struct v4l2_frmsizeenum *p = arg;
- if (!ops->vidioc_enum_framesizes)
- break;
-
ret = ops->vidioc_enum_framesizes(file, fh, p);
dbgarg(cmd,
"index=%d, pixelformat=%c%c%c%c, type=%d ",
{
struct v4l2_frmivalenum *p = arg;
- if (!ops->vidioc_enum_frameintervals)
- break;
-
ret = ops->vidioc_enum_frameintervals(file, fh, p);
dbgarg(cmd,
"index=%d, pixelformat=%d, width=%d, height=%d, type=%d ",
{
struct v4l2_dv_enum_preset *p = arg;
- if (!ops->vidioc_enum_dv_presets)
- break;
-
ret = ops->vidioc_enum_dv_presets(file, fh, p);
if (!ret)
dbgarg(cmd,
{
struct v4l2_dv_preset *p = arg;
- if (!ops->vidioc_s_dv_preset)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
-
dbgarg(cmd, "preset=%d\n", p->preset);
ret = ops->vidioc_s_dv_preset(file, fh, p);
break;
{
struct v4l2_dv_preset *p = arg;
- if (!ops->vidioc_g_dv_preset)
- break;
-
ret = ops->vidioc_g_dv_preset(file, fh, p);
if (!ret)
dbgarg(cmd, "preset=%d\n", p->preset);
{
struct v4l2_dv_preset *p = arg;
- if (!ops->vidioc_query_dv_preset)
- break;
-
ret = ops->vidioc_query_dv_preset(file, fh, p);
if (!ret)
dbgarg(cmd, "preset=%d\n", p->preset);
{
struct v4l2_dv_timings *p = arg;
- if (!ops->vidioc_s_dv_timings)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
-
switch (p->type) {
case V4L2_DV_BT_656_1120:
dbgarg2("bt-656/1120:interlaced=%d, pixelclock=%lld,"
{
struct v4l2_dv_timings *p = arg;
- if (!ops->vidioc_g_dv_timings)
- break;
-
ret = ops->vidioc_g_dv_timings(file, fh, p);
if (!ret) {
switch (p->type) {
{
struct v4l2_event *ev = arg;
- if (!ops->vidioc_subscribe_event)
- break;
-
ret = v4l2_event_dequeue(fh, ev, file->f_flags & O_NONBLOCK);
if (ret < 0) {
dbgarg(cmd, "no pending events?");
{
struct v4l2_event_subscription *sub = arg;
- if (!ops->vidioc_subscribe_event)
- break;
-
ret = ops->vidioc_subscribe_event(fh, sub);
if (ret < 0) {
dbgarg(cmd, "failed, ret=%ld", ret);
{
struct v4l2_event_subscription *sub = arg;
- if (!ops->vidioc_unsubscribe_event)
- break;
-
ret = ops->vidioc_unsubscribe_event(fh, sub);
if (ret < 0) {
dbgarg(cmd, "failed, ret=%ld", ret);
{
struct v4l2_create_buffers *create = arg;
- if (!ops->vidioc_create_bufs)
- break;
- if (ret_prio) {
- ret = ret_prio;
- break;
- }
ret = check_fmt(ops, create->format.type);
if (ret)
break;
{
struct v4l2_buffer *b = arg;
- if (!ops->vidioc_prepare_buf)
- break;
ret = check_fmt(ops, b->type);
if (ret)
break;
default:
if (!ops->vidioc_default)
break;
- ret = ops->vidioc_default(file, fh, ret_prio >= 0, cmd, arg);
+ ret = ops->vidioc_default(file, fh, use_fh_prio ?
+ v4l2_prio_check(vfd->prio, vfh->prio) >= 0 : 0,
+ cmd, arg);
break;
} /* switch */
return ret;
}
EXPORT_SYMBOL_GPL(v4l2_m2m_querybuf);
+/*
+ * v4l2_m2m_expbuf() - multi-queue-not-aware EXPBUF
+ */
+int v4l2_m2m_expbuf(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
+ struct v4l2_exportbuffer *eb)
+{
+ struct vb2_queue *vq;
+
+ /* Below code is written only for FIMC CAPTURE for now */
+ vq = v4l2_m2m_get_vq(m2m_ctx, V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
+ return vb2_expbuf(vq, eb);
+}
+EXPORT_SYMBOL_GPL(v4l2_m2m_expbuf);
/**
* v4l2_m2m_qbuf() - enqueue a source or destination buffer, depending on
case V4L2_MEMORY_OVERLAY:
b->m.offset = vb->boff;
break;
+ case V4L2_MEMORY_DMABUF:
+ /* DMABUF is not handled in videobuf framework */
+ break;
}
b->flags = 0;
break;
case V4L2_MEMORY_USERPTR:
case V4L2_MEMORY_OVERLAY:
+ case V4L2_MEMORY_DMABUF:
/* nothing */
break;
}
}
}
+/**
+ * __vb2_plane_dmabuf_put() - release memory associated with
+ * a DMABUF shared plane
+ */
+static void __vb2_plane_dmabuf_put(struct vb2_queue *q, struct vb2_plane *p)
+{
+ if (!p->mem_priv)
+ return;
+
+ if (p->dbuf_mapped)
+ call_memop(q, unmap_dmabuf, p->mem_priv);
+
+ call_memop(q, detach_dmabuf, p->mem_priv);
+ dma_buf_put(p->dbuf);
+ memset(p, 0, sizeof *p);
+}
+
+/**
+ * __vb2_buf_dmabuf_put() - release memory associated with
+ * a DMABUF shared buffer
+ */
+static void __vb2_buf_dmabuf_put(struct vb2_buffer *vb)
+{
+ struct vb2_queue *q = vb->vb2_queue;
+ unsigned int plane;
+
+ for (plane = 0; plane < vb->num_planes; ++plane)
+ __vb2_plane_dmabuf_put(q, &vb->planes[plane]);
+}
+
/**
* __setup_offsets() - setup unique offsets ("cookies") for every plane in
* every buffer on the queue
/* Free MMAP buffers or release USERPTR buffers */
if (q->memory == V4L2_MEMORY_MMAP)
__vb2_buf_mem_free(vb);
+ else if (q->memory == V4L2_MEMORY_DMABUF)
+ __vb2_buf_dmabuf_put(vb);
else
__vb2_buf_userptr_put(vb);
}
*/
memcpy(b->m.planes, vb->v4l2_planes,
b->length * sizeof(struct v4l2_plane));
+
+ if (q->memory == V4L2_MEMORY_DMABUF) {
+ unsigned int plane;
+ for (plane = 0; plane < vb->num_planes; ++plane)
+ b->m.planes[plane].m.fd = 0;
+ }
} else {
/*
* We use length and offset in v4l2_planes array even for
b->m.offset = vb->v4l2_planes[0].m.mem_offset;
else if (q->memory == V4L2_MEMORY_USERPTR)
b->m.userptr = vb->v4l2_planes[0].m.userptr;
+ else if (q->memory == V4L2_MEMORY_DMABUF)
+ b->m.fd = 0;
}
/*
return 0;
}
+/**
+ * __verify_dmabuf_ops() - verify that all memory operations required for
+ * DMABUF queue type have been provided
+ */
+static int __verify_dmabuf_ops(struct vb2_queue *q)
+{
+ if (!(q->io_modes & VB2_DMABUF) || !q->mem_ops->attach_dmabuf ||
+ !q->mem_ops->detach_dmabuf || !q->mem_ops->map_dmabuf ||
+ !q->mem_ops->unmap_dmabuf)
+ return -EINVAL;
+
+ return 0;
+}
+
/**
* vb2_reqbufs() - Initiate streaming
* @q: videobuf2 queue
return -EBUSY;
}
- if (req->memory != V4L2_MEMORY_MMAP
- && req->memory != V4L2_MEMORY_USERPTR) {
+ if (req->memory != V4L2_MEMORY_MMAP &&
+ req->memory != V4L2_MEMORY_DMABUF &&
+ req->memory != V4L2_MEMORY_USERPTR) {
dprintk(1, "reqbufs: unsupported memory type\n");
return -EINVAL;
}
return -EINVAL;
}
+ if (req->memory == V4L2_MEMORY_DMABUF && __verify_dmabuf_ops(q)) {
+ dprintk(1, "reqbufs: DMABUF for current setup unsupported\n");
+ return -EINVAL;
+ }
+
if (req->count == 0 || q->num_buffers != 0 || q->memory != req->memory) {
/*
* We already have buffers allocated, so first check if they
return -EBUSY;
}
- if (create->memory != V4L2_MEMORY_MMAP
- && create->memory != V4L2_MEMORY_USERPTR) {
+ if (create->memory != V4L2_MEMORY_MMAP &&
+ create->memory != V4L2_MEMORY_USERPTR &&
+ create->memory != V4L2_MEMORY_DMABUF) {
dprintk(1, "%s(): unsupported memory type\n", __func__);
return -EINVAL;
}
return -EINVAL;
}
+ if (create->memory == V4L2_MEMORY_DMABUF && __verify_dmabuf_ops(q)) {
+ dprintk(1, "%s(): DMABUF for current setup unsupported\n", __func__);
+ return -EINVAL;
+ }
+
if (q->num_buffers == VIDEO_MAX_FRAME) {
dprintk(1, "%s(): maximum number of buffers already allocated\n",
__func__);
{
struct vb2_queue *q = vb->vb2_queue;
unsigned long flags;
+ unsigned int plane;
if (vb->state != VB2_BUF_STATE_ACTIVE)
return;
dprintk(4, "Done processing on buffer %d, state: %d\n",
vb->v4l2_buf.index, vb->state);
+ /* sync buffers */
+ for (plane = 0; plane < vb->num_planes; ++plane)
+ call_memop(q, finish, vb->planes[plane].mem_priv);
+
/* Add the buffer to the done buffers list */
spin_lock_irqsave(&q->done_lock, flags);
vb->state = state;
b->m.planes[plane].length;
}
}
+ if (b->memory == V4L2_MEMORY_DMABUF) {
+ for (plane = 0; plane < vb->num_planes; ++plane) {
+ v4l2_planes[plane].bytesused =
+ b->m.planes[plane].bytesused;
+ v4l2_planes[plane].m.fd =
+ b->m.planes[plane].m.fd;
+ }
+ }
} else {
/*
* Single-planar buffers do not use planes array,
v4l2_planes[0].m.userptr = b->m.userptr;
v4l2_planes[0].length = b->length;
}
+
+ if (b->memory == V4L2_MEMORY_DMABUF)
+ v4l2_planes[0].m.fd = b->m.fd;
+
}
vb->v4l2_buf.field = b->field;
return __fill_vb2_buffer(vb, b, vb->v4l2_planes);
}
+/**
+ * __qbuf_dmabuf() - handle qbuf of a DMABUF buffer
+ */
+static int __qbuf_dmabuf(struct vb2_buffer *vb, const struct v4l2_buffer *b)
+{
+ struct v4l2_plane planes[VIDEO_MAX_PLANES];
+ struct vb2_queue *q = vb->vb2_queue;
+ void *mem_priv;
+ unsigned int plane;
+ int ret;
+ int write = !V4L2_TYPE_IS_OUTPUT(q->type);
+
+ /* Verify and copy relevant information provided by the userspace */
+ ret = __fill_vb2_buffer(vb, b, planes);
+ if (ret)
+ return ret;
+
+ for (plane = 0; plane < vb->num_planes; ++plane) {
+ struct dma_buf *dbuf = dma_buf_get(planes[plane].m.fd);
+
+ if (IS_ERR_OR_NULL(dbuf)) {
+ dprintk(1, "qbuf: invalid dmabuf fd for "
+ "plane %d\n", plane);
+ ret = -EINVAL;
+ goto err;
+ }
+
+ /* Skip the plane if already verified */
+ if (dbuf == vb->planes[plane].dbuf) {
+ planes[plane].length = dbuf->size;
+ dma_buf_put(dbuf);
+ continue;
+ }
+
+ dprintk(3, "qbuf: buffer description for plane %d changed, "
+ "reattaching dma buf\n", plane);
+
+ /* Release previously acquired memory if present */
+ __vb2_plane_dmabuf_put(q, &vb->planes[plane]);
+
+ /* Acquire each plane's memory */
+ mem_priv = call_memop(q, attach_dmabuf, q->alloc_ctx[plane],
+ dbuf, q->plane_sizes[plane], write);
+ if (IS_ERR(mem_priv)) {
+ dprintk(1, "qbuf: failed acquiring dmabuf "
+ "memory for plane %d\n", plane);
+ ret = PTR_ERR(mem_priv);
+ goto err;
+ }
+
+ planes[plane].length = dbuf->size;
+ vb->planes[plane].dbuf = dbuf;
+ vb->planes[plane].mem_priv = mem_priv;
+ }
+
+ /* TODO: This pins the buffer(s) with dma_buf_map_attachment()).. but
+ * really we want to do this just before the DMA, not while queueing
+ * the buffer(s)..
+ */
+ for (plane = 0; plane < vb->num_planes; ++plane) {
+ ret = call_memop(q, map_dmabuf, vb->planes[plane].mem_priv);
+ if (ret) {
+ dprintk(1, "qbuf: failed mapping dmabuf "
+ "memory for plane %d\n", plane);
+ goto err;
+ }
+ vb->planes[plane].dbuf_mapped = 1;
+ }
+
+ /*
+ * Call driver-specific initialization on the newly acquired buffer,
+ * if provided.
+ */
+ ret = call_qop(q, buf_init, vb);
+ if (ret) {
+ dprintk(1, "qbuf: buffer initialization failed\n");
+ goto err;
+ }
+
+ /*
+ * Now that everything is in order, copy relevant information
+ * provided by userspace.
+ */
+ for (plane = 0; plane < vb->num_planes; ++plane)
+ vb->v4l2_planes[plane] = planes[plane];
+
+ return 0;
+err:
+ /* In case of errors, release planes that were already acquired */
+ __vb2_buf_dmabuf_put(vb);
+
+ return ret;
+}
+
/**
* __enqueue_in_driver() - enqueue a vb2_buffer in driver for processing
*/
static void __enqueue_in_driver(struct vb2_buffer *vb)
{
struct vb2_queue *q = vb->vb2_queue;
+ unsigned int plane;
vb->state = VB2_BUF_STATE_ACTIVE;
atomic_inc(&q->queued_count);
+
+ /* sync buffers */
+ for (plane = 0; plane < vb->num_planes; ++plane)
+ call_memop(q, prepare, vb->planes[plane].mem_priv);
+
q->ops->buf_queue(vb);
}
case V4L2_MEMORY_USERPTR:
ret = __qbuf_userptr(vb, b);
break;
+ case V4L2_MEMORY_DMABUF:
+ ret = __qbuf_dmabuf(vb, b);
+ break;
default:
WARN(1, "Invalid queue type\n");
ret = -EINVAL;
return ret;
}
+ /* TODO: this unpins the buffer(dma_buf_unmap_attachment()).. but
+ * really we want to do this just after DMA, not when the
+ * buffer is dequeued..
+ */
+ if (q->memory == V4L2_MEMORY_DMABUF) {
+ unsigned int i;
+
+ for (i = 0; i < vb->num_planes; ++i) {
+ call_memop(q, unmap_dmabuf, vb->planes[i].mem_priv);
+ vb->planes[i].dbuf_mapped = 0;
+ }
+ }
+
switch (vb->state) {
case VB2_BUF_STATE_DONE:
dprintk(3, "dqbuf: Returning done buffer\n");
return -EINVAL;
}
+/**
+ * vb2_expbuf() - Export a buffer as a file descriptor
+ * @q: videobuf2 queue
+ * @eb: export buffer structure passed from userspace to vidioc_expbuf
+ * handler in driver
+ *
+ * The return values from this function are intended to be directly returned
+ * from vidioc_expbuf handler in driver.
+ */
+int vb2_expbuf(struct vb2_queue *q, struct v4l2_exportbuffer *eb)
+{
+ struct vb2_buffer *vb = NULL;
+ struct vb2_plane *vb_plane;
+ unsigned int buffer, plane;
+ int ret;
+ struct dma_buf *dbuf;
+
+ if (q->memory != V4L2_MEMORY_MMAP) {
+ dprintk(1, "Queue is not currently set up for mmap\n");
+ return -EINVAL;
+ }
+
+ if (!q->mem_ops->get_dmabuf) {
+ dprintk(1, "Queue does not support DMA buffer exporting\n");
+ return -EINVAL;
+ }
+
+ if (eb->flags & ~O_CLOEXEC) {
+ dprintk(1, "Queue does support only O_CLOEXEC flag\n");
+ return -EINVAL;
+ }
+
+ /*
+ * Find the plane corresponding to the offset passed by userspace.
+ */
+ ret = __find_plane_by_offset(q, eb->mem_offset, &buffer, &plane);
+ if (ret) {
+ dprintk(1, "invalid offset %u\n", eb->mem_offset);
+ return ret;
+ }
+
+ vb = q->bufs[buffer];
+ vb_plane = &vb->planes[plane];
+
+ dbuf = call_memop(q, get_dmabuf, vb_plane->mem_priv);
+ if (IS_ERR_OR_NULL(dbuf)) {
+ dprintk(1, "Failed to export buffer %d, plane %d\n",
+ buffer, plane);
+ return -EINVAL;
+ }
+
+ ret = dma_buf_fd(dbuf, eb->flags);
+ if (ret < 0) {
+ dprintk(3, "buffer %d, plane %d failed to export (%d)\n",
+ buffer, plane, ret);
+ dma_buf_put(dbuf);
+ return ret;
+ }
+
+ dprintk(3, "buffer %d, plane %d exported as %d descriptor\n",
+ buffer, plane, ret);
+ eb->fd = ret;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(vb2_expbuf);
+
/**
* vb2_mmap() - map video buffers into application address space
* @q: videobuf2 queue
* the Free Software Foundation.
*/
+#include <linux/dma-buf.h>
#include <linux/module.h>
+#include <linux/scatterlist.h>
+#include <linux/sched.h>
#include <linux/slab.h>
#include <linux/dma-mapping.h>
};
struct vb2_dc_buf {
- struct vb2_dc_conf *conf;
+ struct device *dev;
void *vaddr;
- dma_addr_t dma_addr;
unsigned long size;
- struct vm_area_struct *vma;
- atomic_t refcount;
+ dma_addr_t dma_addr;
+ enum dma_data_direction dma_dir;
+ struct sg_table *dma_sgt;
+
+ /* MMAP related */
struct vb2_vmarea_handler handler;
+ atomic_t refcount;
+ struct sg_table *sgt_base;
+
+ /* USERPTR related */
+ struct vm_area_struct *vma;
+
+ /* DMABUF related */
+ struct dma_buf_attachment *db_attach;
};
-static void vb2_dma_contig_put(void *buf_priv);
+/*********************************************/
+/* scatterlist table functions */
+/*********************************************/
+
+
+static void vb2_dc_sgt_foreach_page(struct sg_table *sgt,
+ void (*cb)(struct page *pg))
+{
+ struct scatterlist *s;
+ unsigned int i;
+
+ for_each_sg(sgt->sgl, s, sgt->orig_nents, i) {
+ struct page *page = sg_page(s);
+ unsigned int n_pages = PAGE_ALIGN(s->offset + s->length)
+ >> PAGE_SHIFT;
+ unsigned int j;
+
+ for (j = 0; j < n_pages; ++j, ++page)
+ cb(page);
+ }
+}
+
+static unsigned long vb2_dc_get_contiguous_size(struct sg_table *sgt)
+{
+ struct scatterlist *s;
+ dma_addr_t expected = sg_dma_address(sgt->sgl);
+ unsigned int i;
+ unsigned long size = 0;
+
+ for_each_sg(sgt->sgl, s, sgt->nents, i) {
+ if (sg_dma_address(s) != expected)
+ break;
+ expected = sg_dma_address(s) + sg_dma_len(s);
+ size += sg_dma_len(s);
+ }
+ return size;
+}
+
+/*********************************************/
+/* callbacks for all buffers */
+/*********************************************/
+
+static void *vb2_dc_cookie(void *buf_priv)
+{
+ struct vb2_dc_buf *buf = buf_priv;
+
+ return &buf->dma_addr;
+}
+
+static void *vb2_dc_vaddr(void *buf_priv)
+{
+ struct vb2_dc_buf *buf = buf_priv;
+
+ return buf->vaddr;
+}
+
+static unsigned int vb2_dc_num_users(void *buf_priv)
+{
+ struct vb2_dc_buf *buf = buf_priv;
+
+ return atomic_read(&buf->refcount);
+}
+
+static void vb2_dc_prepare(void *buf_priv)
+{
+ struct vb2_dc_buf *buf = buf_priv;
+ struct sg_table *sgt = buf->dma_sgt;
+
+ /* DMABUF exporter will flush the cache for us */
+ if (!sgt || buf->db_attach)
+ return;
+
+ dma_sync_sg_for_device(buf->dev, sgt->sgl, sgt->nents, buf->dma_dir);
+}
+
+static void vb2_dc_finish(void *buf_priv)
+{
+ struct vb2_dc_buf *buf = buf_priv;
+ struct sg_table *sgt = buf->dma_sgt;
+
+ /* DMABUF exporter will flush the cache for us */
+ if (!sgt || buf->db_attach)
+ return;
+
+ dma_sync_sg_for_cpu(buf->dev, sgt->sgl, sgt->nents, buf->dma_dir);
+}
+
+/*********************************************/
+/* callbacks for MMAP buffers */
+/*********************************************/
+
+static void vb2_dc_put(void *buf_priv)
+{
+ struct vb2_dc_buf *buf = buf_priv;
+
+ if (!atomic_dec_and_test(&buf->refcount))
+ return;
+
+ if (buf->sgt_base) {
+ sg_free_table(buf->sgt_base);
+ kfree(buf->sgt_base);
+ }
+ dma_free_coherent(buf->dev, buf->size, buf->vaddr, buf->dma_addr);
+ kfree(buf);
+}
-static void *vb2_dma_contig_alloc(void *alloc_ctx, unsigned long size)
+static void *vb2_dc_alloc(void *alloc_ctx, unsigned long size)
{
struct vb2_dc_conf *conf = alloc_ctx;
+ struct device *dev = conf->dev;
struct vb2_dc_buf *buf;
buf = kzalloc(sizeof *buf, GFP_KERNEL);
if (!buf)
return ERR_PTR(-ENOMEM);
- buf->vaddr = dma_alloc_coherent(conf->dev, size, &buf->dma_addr,
- GFP_KERNEL);
+ buf->vaddr = dma_alloc_coherent(dev, size, &buf->dma_addr, GFP_KERNEL);
if (!buf->vaddr) {
- dev_err(conf->dev, "dma_alloc_coherent of size %ld failed\n",
- size);
+ dev_err(dev, "dma_alloc_coherent of size %ld failed\n", size);
kfree(buf);
return ERR_PTR(-ENOMEM);
}
- buf->conf = conf;
+ buf->dev = dev;
buf->size = size;
buf->handler.refcount = &buf->refcount;
- buf->handler.put = vb2_dma_contig_put;
+ buf->handler.put = vb2_dc_put;
buf->handler.arg = buf;
atomic_inc(&buf->refcount);
return buf;
}
-static void vb2_dma_contig_put(void *buf_priv)
+static int vb2_dc_mmap(void *buf_priv, struct vm_area_struct *vma)
{
struct vb2_dc_buf *buf = buf_priv;
+ int ret;
- if (atomic_dec_and_test(&buf->refcount)) {
- dma_free_coherent(buf->conf->dev, buf->size, buf->vaddr,
- buf->dma_addr);
- kfree(buf);
+ if (!buf) {
+ printk(KERN_ERR "No buffer to map\n");
+ return -EINVAL;
+ }
+
+ /*
+ * dma_mmap_* uses vm_pgoff as in-buffer offset, but we want to
+ * map whole buffer
+ */
+ vma->vm_pgoff = 0;
+
+ ret = dma_mmap_coherent(buf->dev, vma, buf->vaddr,
+ buf->dma_addr, buf->size);
+
+ if (ret) {
+ printk(KERN_ERR "Remapping memory failed, error: %d\n", ret);
+ return ret;
}
+
+ vma->vm_flags |= VM_DONTEXPAND | VM_RESERVED;
+ vma->vm_private_data = &buf->handler;
+ vma->vm_ops = &vb2_common_vm_ops;
+
+ vma->vm_ops->open(vma);
+
+ printk(KERN_DEBUG "%s: mapped dma addr 0x%08lx at 0x%08lx, size %ld\n",
+ __func__, (unsigned long)buf->dma_addr, vma->vm_start,
+ buf->size);
+
+ return 0;
}
-static void *vb2_dma_contig_cookie(void *buf_priv)
+/*********************************************/
+/* DMABUF ops for exporters */
+/*********************************************/
+
+struct vb2_dc_attachment {
+ struct sg_table sgt;
+ enum dma_data_direction dir;
+};
+
+static int vb2_dc_dmabuf_ops_attach(struct dma_buf *dbuf, struct device *dev,
+ struct dma_buf_attachment *dbuf_attach)
{
- struct vb2_dc_buf *buf = buf_priv;
+ struct vb2_dc_attachment *attach;
+ unsigned int i;
+ struct scatterlist *rd, *wr;
+ struct sg_table *sgt;
+ struct vb2_dc_buf *buf = dbuf->priv;
+ int ret;
- return &buf->dma_addr;
+ attach = kzalloc(sizeof *attach, GFP_KERNEL);
+ if (!attach)
+ return -ENOMEM;
+
+ sgt = &attach->sgt;
+ /* Copy the buf->base_sgt scatter list to the attachment, as we can't
+ * map the same scatter list to multiple attachments at the same time.
+ */
+ ret = sg_alloc_table(sgt, buf->sgt_base->orig_nents, GFP_KERNEL);
+ if (ret) {
+ kfree(attach);
+ return -ENOMEM;
+ }
+
+ rd = buf->sgt_base->sgl;
+ wr = sgt->sgl;
+ for (i = 0; i < sgt->orig_nents; ++i) {
+ sg_set_page(wr, sg_page(rd), rd->length, rd->offset);
+ rd = sg_next(rd);
+ wr = sg_next(wr);
+ }
+
+ attach->dir = DMA_NONE;
+ dbuf_attach->priv = attach;
+
+ return 0;
}
-static void *vb2_dma_contig_vaddr(void *buf_priv)
+static void vb2_dc_dmabuf_ops_detach(struct dma_buf *dbuf,
+ struct dma_buf_attachment *db_attach)
{
- struct vb2_dc_buf *buf = buf_priv;
- if (!buf)
- return NULL;
+ struct vb2_dc_attachment *attach = db_attach->priv;
+ struct sg_table *sgt;
+
+ if (!attach)
+ return;
+
+ sgt = &attach->sgt;
+
+ /* release the scatterlist cache */
+ if (attach->dir != DMA_NONE)
+ dma_unmap_sg(db_attach->dev, sgt->sgl, sgt->orig_nents,
+ attach->dir);
+ sg_free_table(sgt);
+ kfree(attach);
+ db_attach->priv = NULL;
+}
+
+static struct sg_table *vb2_dc_dmabuf_ops_map(
+ struct dma_buf_attachment *db_attach, enum dma_data_direction dir)
+{
+ struct vb2_dc_attachment *attach = db_attach->priv;
+ /* stealing dmabuf mutex to serialize map/unmap operations */
+ struct mutex *lock = &db_attach->dmabuf->lock;
+ struct sg_table *sgt;
+ int ret;
+
+ mutex_lock(lock);
+
+ sgt = &attach->sgt;
+ /* return previously mapped sg table */
+ if (attach->dir == dir) {
+ mutex_unlock(lock);
+ return sgt;
+ }
+
+ /* release any previous cache */
+ if (attach->dir != DMA_NONE) {
+ dma_unmap_sg(db_attach->dev, sgt->sgl, sgt->orig_nents,
+ attach->dir);
+ attach->dir = DMA_NONE;
+ }
+
+ /* mapping to the client with new direction */
+ ret = dma_map_sg(db_attach->dev, sgt->sgl, sgt->orig_nents, dir);
+ if (ret <= 0) {
+ printk(KERN_ERR "failed to map scatterlist\n");
+ mutex_unlock(lock);
+ return ERR_PTR(-EIO);
+ }
+
+ attach->dir = dir;
+
+ mutex_unlock(lock);
+
+ return sgt;
+}
+
+static void vb2_dc_dmabuf_ops_unmap(struct dma_buf_attachment *db_attach,
+ struct sg_table *sgt, enum dma_data_direction dir)
+{
+ /* nothing to be done here */
+}
+
+static void vb2_dc_dmabuf_ops_release(struct dma_buf *dbuf)
+{
+ /* drop reference obtained in vb2_dc_get_dmabuf */
+ vb2_dc_put(dbuf->priv);
+}
+
+static void *vb2_dc_dmabuf_ops_kmap(struct dma_buf *dbuf, unsigned long pgnum)
+{
+ struct vb2_dc_buf *buf = dbuf->priv;
+
+ return buf->vaddr + pgnum * PAGE_SIZE;
+}
+
+static void *vb2_dc_dmabuf_ops_vmap(struct dma_buf *dbuf)
+{
+ struct vb2_dc_buf *buf = dbuf->priv;
return buf->vaddr;
}
-static unsigned int vb2_dma_contig_num_users(void *buf_priv)
+/* a dummy function to support the mmap functionality for now */
+static void *vb2_dc_dmabuf_ops_mmap(struct dma_buf *dbuf)
{
- struct vb2_dc_buf *buf = buf_priv;
+ /* do nothing */
+ return NULL;
+}
- return atomic_read(&buf->refcount);
+static struct dma_buf_ops vb2_dc_dmabuf_ops = {
+ .attach = vb2_dc_dmabuf_ops_attach,
+ .detach = vb2_dc_dmabuf_ops_detach,
+ .map_dma_buf = vb2_dc_dmabuf_ops_map,
+ .unmap_dma_buf = vb2_dc_dmabuf_ops_unmap,
+ .kmap = vb2_dc_dmabuf_ops_kmap,
+ .kmap_atomic = vb2_dc_dmabuf_ops_kmap,
+ .vmap = vb2_dc_dmabuf_ops_vmap,
+ .mmap = vb2_dc_dmabuf_ops_mmap,
+ .release = vb2_dc_dmabuf_ops_release,
+};
+
+static struct sg_table *vb2_dc_get_base_sgt(struct vb2_dc_buf *buf)
+{
+ int ret;
+ struct sg_table *sgt;
+
+ sgt = kmalloc(sizeof *sgt, GFP_KERNEL);
+ if (!sgt) {
+ dev_err(buf->dev, "failed to alloc sg table\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ ret = dma_get_sgtable(buf->dev, sgt, buf->vaddr, buf->dma_addr,
+ buf->size);
+ if (ret < 0) {
+ dev_err(buf->dev, "failed to get scatterlist from DMA API\n");
+ kfree(sgt);
+ return ERR_PTR(ret);
+ }
+
+ return sgt;
}
-static int vb2_dma_contig_mmap(void *buf_priv, struct vm_area_struct *vma)
+static struct dma_buf *vb2_dc_get_dmabuf(void *buf_priv)
{
struct vb2_dc_buf *buf = buf_priv;
+ struct dma_buf *dbuf;
+ struct sg_table *sgt = buf->sgt_base;
- if (!buf) {
- printk(KERN_ERR "No buffer to map\n");
- return -EINVAL;
+ if (!sgt)
+ sgt = vb2_dc_get_base_sgt(buf);
+ if (WARN_ON(IS_ERR(sgt)))
+ return NULL;
+
+ /* cache base sgt for future use */
+ buf->sgt_base = sgt;
+
+ dbuf = dma_buf_export(buf, &vb2_dc_dmabuf_ops, buf->size, 0);
+ if (IS_ERR(dbuf))
+ return NULL;
+
+ /* dmabuf keeps reference to vb2 buffer */
+ atomic_inc(&buf->refcount);
+
+ return dbuf;
+}
+
+/*********************************************/
+/* callbacks for USERPTR buffers */
+/*********************************************/
+
+static inline int vma_is_io(struct vm_area_struct *vma)
+{
+ return !!(vma->vm_flags & (VM_IO | VM_PFNMAP));
+}
+
+static int vb2_dc_get_user_pages(unsigned long start, struct page **pages,
+ int n_pages, struct vm_area_struct *vma, int write)
+{
+ if (vma_is_io(vma)) {
+ unsigned int i;
+
+ for (i = 0; i < n_pages; ++i, start += PAGE_SIZE) {
+ unsigned long pfn;
+ int ret = follow_pfn(vma, start, &pfn);
+
+ if (ret) {
+ printk(KERN_ERR "no page for address %lu\n",
+ start);
+ return ret;
+ }
+ pages[i] = pfn_to_page(pfn);
+ }
+ } else {
+ int n;
+
+ n = get_user_pages(current, current->mm, start & PAGE_MASK,
+ n_pages, write, 1, pages, NULL);
+ /* negative error means that no page was pinned */
+ n = max(n, 0);
+ if (n != n_pages) {
+ printk(KERN_ERR "got only %d of %d user pages\n",
+ n, n_pages);
+ while (n)
+ put_page(pages[--n]);
+ return -EFAULT;
+ }
}
- return vb2_mmap_pfn_range(vma, buf->dma_addr, buf->size,
- &vb2_common_vm_ops, &buf->handler);
+ return 0;
}
-static void *vb2_dma_contig_get_userptr(void *alloc_ctx, unsigned long vaddr,
- unsigned long size, int write)
+static void vb2_dc_put_dirty_page(struct page *page)
{
+ set_page_dirty_lock(page);
+ put_page(page);
+}
+
+static void vb2_dc_put_userptr(void *buf_priv)
+{
+ struct vb2_dc_buf *buf = buf_priv;
+ struct sg_table *sgt = buf->dma_sgt;
+
+ dma_unmap_sg(buf->dev, sgt->sgl, sgt->orig_nents, buf->dma_dir);
+ if (!vma_is_io(buf->vma))
+ vb2_dc_sgt_foreach_page(sgt, vb2_dc_put_dirty_page);
+
+ sg_free_table(sgt);
+ kfree(sgt);
+ vb2_put_vma(buf->vma);
+ kfree(buf);
+}
+
+static void *vb2_dc_get_userptr(void *alloc_ctx, unsigned long vaddr,
+ unsigned long size, int write)
+{
+ struct vb2_dc_conf *conf = alloc_ctx;
struct vb2_dc_buf *buf;
+ unsigned long start;
+ unsigned long end;
+ unsigned long offset;
+ struct page **pages;
+ int n_pages;
+ int ret = 0;
struct vm_area_struct *vma;
- dma_addr_t dma_addr = 0;
- int ret;
+ struct sg_table *sgt;
+ unsigned long contig_size;
buf = kzalloc(sizeof *buf, GFP_KERNEL);
if (!buf)
return ERR_PTR(-ENOMEM);
- ret = vb2_get_contig_userptr(vaddr, size, &vma, &dma_addr);
+ buf->dev = conf->dev;
+ buf->dma_dir = write ? DMA_FROM_DEVICE : DMA_TO_DEVICE;
+
+ start = vaddr & PAGE_MASK;
+ offset = vaddr & ~PAGE_MASK;
+ end = PAGE_ALIGN(vaddr + size);
+ n_pages = (end - start) >> PAGE_SHIFT;
+
+ pages = kmalloc(n_pages * sizeof pages[0], GFP_KERNEL);
+ if (!pages) {
+ ret = -ENOMEM;
+ printk(KERN_ERR "failed to allocate pages table\n");
+ goto fail_buf;
+ }
+
+ /* current->mm->mmap_sem is taken by videobuf2 core */
+ vma = find_vma(current->mm, vaddr);
+ if (!vma) {
+ printk(KERN_ERR "no vma for address %lu\n", vaddr);
+ ret = -EFAULT;
+ goto fail_pages;
+ }
+
+ if (vma->vm_end < vaddr + size) {
+ printk(KERN_ERR "vma at %lu is too small for %lu bytes\n",
+ vaddr, size);
+ ret = -EFAULT;
+ goto fail_pages;
+ }
+
+ buf->vma = vb2_get_vma(vma);
+ if (!buf->vma) {
+ printk(KERN_ERR "failed to copy vma\n");
+ ret = -ENOMEM;
+ goto fail_pages;
+ }
+
+ /* extract page list from userspace mapping */
+ ret = vb2_dc_get_user_pages(start, pages, n_pages, vma, write);
if (ret) {
- printk(KERN_ERR "Failed acquiring VMA for vaddr 0x%08lx\n",
- vaddr);
- kfree(buf);
- return ERR_PTR(ret);
+ printk(KERN_ERR "failed to get user pages\n");
+ goto fail_vma;
+ }
+
+ sgt = kzalloc(sizeof *sgt, GFP_KERNEL);
+ if (!sgt) {
+ printk(KERN_ERR "failed to allocate sg table\n");
+ ret = -ENOMEM;
+ goto fail_get_user_pages;
+ }
+
+ ret = sg_alloc_table_from_pages(sgt, pages, n_pages,
+ offset, size, GFP_KERNEL);
+ if (ret) {
+ printk(KERN_ERR "failed to initialize sg table\n");
+ goto fail_sgt;
+ }
+
+ /* pages are no longer needed */
+ kfree(pages);
+ pages = NULL;
+
+ sgt->nents = dma_map_sg(buf->dev, sgt->sgl, sgt->orig_nents,
+ buf->dma_dir);
+ if (sgt->nents <= 0) {
+ printk(KERN_ERR "failed to map scatterlist\n");
+ ret = -EIO;
+ goto fail_sgt_init;
+ }
+
+ contig_size = vb2_dc_get_contiguous_size(sgt);
+ if (contig_size < size) {
+ printk(KERN_ERR "contiguous mapping is too small %lu/%lu\n",
+ contig_size, size);
+ ret = -EFAULT;
+ goto fail_map_sg;
}
+ buf->dma_addr = sg_dma_address(sgt->sgl);
buf->size = size;
- buf->dma_addr = dma_addr;
- buf->vma = vma;
+ buf->dma_sgt = sgt;
return buf;
+
+fail_map_sg:
+ dma_unmap_sg(buf->dev, sgt->sgl, sgt->orig_nents, buf->dma_dir);
+
+fail_sgt_init:
+ if (!vma_is_io(buf->vma))
+ vb2_dc_sgt_foreach_page(sgt, put_page);
+ sg_free_table(sgt);
+
+fail_sgt:
+ kfree(sgt);
+
+fail_get_user_pages:
+ if (pages && !vma_is_io(buf->vma))
+ while (n_pages)
+ put_page(pages[--n_pages]);
+
+fail_vma:
+ vb2_put_vma(buf->vma);
+
+fail_pages:
+ kfree(pages); /* kfree is NULL-proof */
+
+fail_buf:
+ kfree(buf);
+
+ return ERR_PTR(ret);
}
-static void vb2_dma_contig_put_userptr(void *mem_priv)
+/*********************************************/
+/* callbacks for DMABUF buffers */
+/*********************************************/
+
+static int vb2_dc_map_dmabuf(void *mem_priv)
{
struct vb2_dc_buf *buf = mem_priv;
+ struct sg_table *sgt;
+ unsigned long contig_size;
- if (!buf)
+ if (WARN_ON(!buf->db_attach)) {
+ printk(KERN_ERR "trying to pin a non attached buffer\n");
+ return -EINVAL;
+ }
+
+ if (WARN_ON(buf->dma_sgt)) {
+ printk(KERN_ERR "dmabuf buffer is already pinned\n");
+ return 0;
+ }
+
+ /* get the associated scatterlist for this buffer */
+ sgt = dma_buf_map_attachment(buf->db_attach, buf->dma_dir);
+ if (IS_ERR_OR_NULL(sgt)) {
+ printk(KERN_ERR "Error getting dmabuf scatterlist\n");
+ return -EINVAL;
+ }
+
+ /* checking if dmabuf is big enough to store contiguous chunk */
+ contig_size = vb2_dc_get_contiguous_size(sgt);
+ if (contig_size < buf->size) {
+ printk(KERN_ERR "contiguous chunk is too small %lu/%lu b\n",
+ contig_size, buf->size);
+ dma_buf_unmap_attachment(buf->db_attach, sgt, buf->dma_dir);
+ return -EFAULT;
+ }
+
+ buf->dma_addr = sg_dma_address(sgt->sgl);
+ buf->dma_sgt = sgt;
+
+ return 0;
+}
+
+static void vb2_dc_unmap_dmabuf(void *mem_priv)
+{
+ struct vb2_dc_buf *buf = mem_priv;
+ struct sg_table *sgt = buf->dma_sgt;
+
+ if (WARN_ON(!buf->db_attach)) {
+ printk(KERN_ERR "trying to unpin a not attached buffer\n");
return;
+ }
- vb2_put_vma(buf->vma);
+ if (WARN_ON(!sgt)) {
+ printk(KERN_ERR "dmabuf buffer is already unpinned\n");
+ return;
+ }
+
+ dma_buf_unmap_attachment(buf->db_attach, sgt, buf->dma_dir);
+
+ buf->dma_addr = 0;
+ buf->dma_sgt = NULL;
+}
+
+static void vb2_dc_detach_dmabuf(void *mem_priv)
+{
+ struct vb2_dc_buf *buf = mem_priv;
+
+ /* if vb2 works correctly you should never detach mapped buffer */
+ if (WARN_ON(buf->dma_addr))
+ vb2_dc_unmap_dmabuf(buf);
+
+ /* detach this attachment */
+ dma_buf_detach(buf->db_attach->dmabuf, buf->db_attach);
kfree(buf);
}
+static void *vb2_dc_attach_dmabuf(void *alloc_ctx, struct dma_buf *dbuf,
+ unsigned long size, int write)
+{
+ struct vb2_dc_conf *conf = alloc_ctx;
+ struct vb2_dc_buf *buf;
+ struct dma_buf_attachment *dba;
+
+ if (dbuf->size < size)
+ return ERR_PTR(-EFAULT);
+
+ buf = kzalloc(sizeof *buf, GFP_KERNEL);
+ if (!buf)
+ return ERR_PTR(-ENOMEM);
+
+ buf->dev = conf->dev;
+ /* create attachment for the dmabuf with the user device */
+ dba = dma_buf_attach(dbuf, buf->dev);
+ if (IS_ERR(dba)) {
+ printk(KERN_ERR "failed to attach dmabuf\n");
+ kfree(buf);
+ return dba;
+ }
+
+ buf->dma_dir = write ? DMA_FROM_DEVICE : DMA_TO_DEVICE;
+ buf->size = size;
+ buf->db_attach = dba;
+
+ return buf;
+}
+
+/*********************************************/
+/* DMA CONTIG exported functions */
+/*********************************************/
+
const struct vb2_mem_ops vb2_dma_contig_memops = {
- .alloc = vb2_dma_contig_alloc,
- .put = vb2_dma_contig_put,
- .cookie = vb2_dma_contig_cookie,
- .vaddr = vb2_dma_contig_vaddr,
- .mmap = vb2_dma_contig_mmap,
- .get_userptr = vb2_dma_contig_get_userptr,
- .put_userptr = vb2_dma_contig_put_userptr,
- .num_users = vb2_dma_contig_num_users,
+ .alloc = vb2_dc_alloc,
+ .put = vb2_dc_put,
+ .get_dmabuf = vb2_dc_get_dmabuf,
+ .cookie = vb2_dc_cookie,
+ .vaddr = vb2_dc_vaddr,
+ .mmap = vb2_dc_mmap,
+ .get_userptr = vb2_dc_get_userptr,
+ .put_userptr = vb2_dc_put_userptr,
+ .prepare = vb2_dc_prepare,
+ .finish = vb2_dc_finish,
+ .map_dmabuf = vb2_dc_map_dmabuf,
+ .unmap_dmabuf = vb2_dc_unmap_dmabuf,
+ .attach_dmabuf = vb2_dc_attach_dmabuf,
+ .detach_dmabuf = vb2_dc_detach_dmabuf,
+ .num_users = vb2_dc_num_users,
};
EXPORT_SYMBOL_GPL(vb2_dma_contig_memops);
}
EXPORT_SYMBOL_GPL(vb2_get_contig_userptr);
-/**
- * vb2_mmap_pfn_range() - map physical pages to userspace
- * @vma: virtual memory region for the mapping
- * @paddr: starting physical address of the memory to be mapped
- * @size: size of the memory to be mapped
- * @vm_ops: vm operations to be assigned to the created area
- * @priv: private data to be associated with the area
- *
- * Returns 0 on success.
- */
-int vb2_mmap_pfn_range(struct vm_area_struct *vma, unsigned long paddr,
- unsigned long size,
- const struct vm_operations_struct *vm_ops,
- void *priv)
-{
- int ret;
-
- size = min_t(unsigned long, vma->vm_end - vma->vm_start, size);
-
- vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
- ret = remap_pfn_range(vma, vma->vm_start, paddr >> PAGE_SHIFT,
- size, vma->vm_page_prot);
- if (ret) {
- printk(KERN_ERR "Remapping memory failed, error: %d\n", ret);
- return ret;
- }
-
- vma->vm_flags |= VM_DONTEXPAND | VM_RESERVED;
- vma->vm_private_data = priv;
- vma->vm_ops = vm_ops;
-
- vma->vm_ops->open(vma);
-
- pr_debug("%s: mapped paddr 0x%08lx at 0x%08lx, size %ld\n",
- __func__, paddr, vma->vm_start, size);
-
- return 0;
-}
-EXPORT_SYMBOL_GPL(vb2_mmap_pfn_range);
-
/**
* vb2_common_vm_open() - increase refcount of the vma
* @vma: virtual memory region for the mapping
unsigned int n_pages;
atomic_t refcount;
struct vb2_vmarea_handler handler;
+ struct dma_buf *dbuf;
};
static void vb2_vmalloc_put(void *buf_priv);
return 0;
}
+/*********************************************/
+/* callbacks for DMABUF buffers */
+/*********************************************/
+
+static int vb2_vmalloc_map_dmabuf(void *mem_priv)
+{
+ struct vb2_vmalloc_buf *buf = mem_priv;
+
+ buf->vaddr = dma_buf_vmap(buf->dbuf);
+
+ return buf->vaddr ? 0 : -EFAULT;
+}
+
+static void vb2_vmalloc_unmap_dmabuf(void *mem_priv)
+{
+ struct vb2_vmalloc_buf *buf = mem_priv;
+
+ dma_buf_vunmap(buf->dbuf, buf->vaddr);
+ buf->vaddr = NULL;
+}
+
+static void vb2_vmalloc_detach_dmabuf(void *mem_priv)
+{
+ struct vb2_vmalloc_buf *buf = mem_priv;
+
+ if (buf->vaddr)
+ dma_buf_vunmap(buf->dbuf, buf->vaddr);
+
+ kfree(buf);
+}
+
+static void *vb2_vmalloc_attach_dmabuf(void *alloc_ctx, struct dma_buf *dbuf,
+ unsigned long size, int write)
+{
+ struct vb2_vmalloc_buf *buf;
+
+ if (dbuf->size < size)
+ return ERR_PTR(-EFAULT);
+
+ buf = kzalloc(sizeof *buf, GFP_KERNEL);
+ if (!buf)
+ return ERR_PTR(-ENOMEM);
+
+ buf->dbuf = dbuf;
+ buf->write = write;
+ buf->size = size;
+
+ return buf;
+}
+
+
const struct vb2_mem_ops vb2_vmalloc_memops = {
.alloc = vb2_vmalloc_alloc,
.put = vb2_vmalloc_put,
.get_userptr = vb2_vmalloc_get_userptr,
.put_userptr = vb2_vmalloc_put_userptr,
+ .map_dmabuf = vb2_vmalloc_map_dmabuf,
+ .unmap_dmabuf = vb2_vmalloc_unmap_dmabuf,
+ .attach_dmabuf = vb2_vmalloc_attach_dmabuf,
+ .detach_dmabuf = vb2_vmalloc_detach_dmabuf,
.vaddr = vb2_vmalloc_vaddr,
.mmap = vb2_vmalloc_mmap,
.num_users = vb2_vmalloc_num_users,
q = &dev->vb_vidq;
memset(q, 0, sizeof(dev->vb_vidq));
q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
- q->io_modes = VB2_MMAP | VB2_USERPTR | VB2_READ;
+ q->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF | VB2_READ;
q->drv_priv = dev;
q->buf_struct_size = sizeof(struct vivi_buffer);
q->ops = &vivi_video_qops;
.ctrl_type = DW_MCI_TYPE_SYNOPSIS,
};
+static unsigned long exynos5250_dwmmc_caps[4] = {
+ MMC_CAP_UHS_DDR50 | MMC_CAP_1_8V_DDR |
+ MMC_CAP_8_BIT_DATA | MMC_CAP_CMD23,
+ MMC_CAP_CMD23,
+ MMC_CAP_CMD23,
+ MMC_CAP_CMD23,
+};
+
static struct dw_mci_drv_data exynos5250_drv_data = {
.ctrl_type = DW_MCI_TYPE_EXYNOS5250,
- .caps = MMC_CAP_UHS_DDR50 | MMC_CAP_1_8V_DDR |
- MMC_CAP_8_BIT_DATA | MMC_CAP_CMD23,
+ .caps = exynos5250_dwmmc_caps,
};
static const struct of_device_id dw_mci_pltfm_match[] = {
if (pending & (SDMMC_IDMAC_INT_TI | SDMMC_IDMAC_INT_RI)) {
mci_writel(host, IDSTS, SDMMC_IDMAC_INT_TI | SDMMC_IDMAC_INT_RI);
mci_writel(host, IDSTS, SDMMC_IDMAC_INT_NI);
- set_bit(EVENT_DATA_COMPLETE, &host->pending_events);
host->dma_ops->complete(host);
}
#endif
{
struct mmc_host *mmc;
struct dw_mci_slot *slot;
+ unsigned int ctrl_id;
mmc = mmc_alloc_host(sizeof(struct dw_mci_slot), &host->dev);
if (!mmc)
if (host->pdata->caps)
mmc->caps = host->pdata->caps;
- mmc->caps |= host->drv_data->caps;
+ if (host->dev.of_node) {
+ ctrl_id = of_alias_get_id(host->dev.of_node, "mshc");
+ if (ctrl_id < 0)
+ ctrl_id = 0;
+ mmc->caps |= host->drv_data->caps[ctrl_id];
+ }
if (host->pdata->caps2)
mmc->caps2 = host->pdata->caps2;
/* dw_mci platform driver data */
struct dw_mci_drv_data {
unsigned long ctrl_type;
- unsigned long caps;
+ unsigned long *caps;
};
#endif /* _DW_MMC_H_ */
{ "w25q32", INFO(0xef4016, 0, 64 * 1024, 64, SECT_4K) },
{ "w25x64", INFO(0xef3017, 0, 64 * 1024, 128, SECT_4K) },
{ "w25q64", INFO(0xef4017, 0, 64 * 1024, 128, SECT_4K) },
+ { "w25q80", INFO(0xef5014, 0, 64 * 1024, 16, SECT_4K) },
/* Catalyst / On Semiconductor -- non-JEDEC */
{ "cat25c11", CAT25_INFO( 16, 8, 16, 1) },
#include <linux/platform_device.h>
#include <linux/pm_runtime.h>
#include <linux/spi/spi.h>
+#include <linux/of.h>
+#include <linux/of_gpio.h>
#include <mach/dma.h>
#include <plat/s3c64xx-spi.h>
+#define MAX_SPI_PORTS 3
+
/* Registers and bit-fields */
#define S3C64XX_SPI_CH_CFG 0x00
#define S3C64XX_SPI_FBCLK_MSK (3<<0)
-#define S3C64XX_SPI_ST_TRLCNTZ(v, i) ((((v) >> (i)->rx_lvl_offset) & \
- (((i)->fifo_lvl_mask + 1))) \
- ? 1 : 0)
-
-#define S3C64XX_SPI_ST_TX_DONE(v, i) (((v) & (1 << (i)->tx_st_done)) ? 1 : 0)
-#define TX_FIFO_LVL(v, i) (((v) >> 6) & (i)->fifo_lvl_mask)
-#define RX_FIFO_LVL(v, i) (((v) >> (i)->rx_lvl_offset) & (i)->fifo_lvl_mask)
+#define FIFO_LVL_MASK(i) ((i)->port_conf->fifo_lvl_mask[i->port_id])
+#define S3C64XX_SPI_ST_TX_DONE(v, i) (((v) & \
+ (1 << (i)->port_conf->tx_st_done)) ? 1 : 0)
+#define TX_FIFO_LVL(v, i) (((v) >> 6) & FIFO_LVL_MASK(i))
+#define RX_FIFO_LVL(v, i) (((v) >> (i)->port_conf->rx_lvl_offset) & \
+ FIFO_LVL_MASK(i))
#define S3C64XX_SPI_MAX_TRAILCNT 0x3ff
#define S3C64XX_SPI_TRAILCNT_OFF 19
unsigned ch;
enum dma_data_direction direction;
enum dma_ch dmach;
+ struct property *dma_prop;
+};
+
+/**
+ * struct s3c64xx_spi_info - SPI Controller hardware info
+ * @fifo_lvl_mask: Bit-mask for {TX|RX}_FIFO_LVL bits in SPI_STATUS register.
+ * @rx_lvl_offset: Bit offset of RX_FIFO_LVL bits in SPI_STATUS regiter.
+ * @tx_st_done: Bit offset of TX_DONE bit in SPI_STATUS regiter.
+ * @high_speed: True, if the controller supports HIGH_SPEED_EN bit.
+ * @clk_from_cmu: True, if the controller does not include a clock mux and
+ * prescaler unit.
+ *
+ * The Samsung s3c64xx SPI controller are used on various Samsung SoC's but
+ * differ in some aspects such as the size of the fifo and spi bus clock
+ * setup. Such differences are specified to the driver using this structure
+ * which is provided as driver data to the driver.
+ */
+struct s3c64xx_spi_port_config {
+ int fifo_lvl_mask[MAX_SPI_PORTS];
+ int rx_lvl_offset;
+ int tx_st_done;
+ bool high_speed;
+ bool clk_from_cmu;
};
/**
struct s3c64xx_spi_dma_data rx_dma;
struct s3c64xx_spi_dma_data tx_dma;
struct samsung_dma_ops *ops;
+ struct s3c64xx_spi_port_config *port_conf;
+ unsigned int port_id;
+ unsigned long gpios[4];
};
static struct s3c2410_dma_client s3c64xx_spi_dma_client = {
static void flush_fifo(struct s3c64xx_spi_driver_data *sdd)
{
- struct s3c64xx_spi_info *sci = sdd->cntrlr_info;
void __iomem *regs = sdd->regs;
unsigned long loops;
u32 val;
loops = msecs_to_loops(1);
do {
val = readl(regs + S3C64XX_SPI_STATUS);
- } while (TX_FIFO_LVL(val, sci) && loops--);
+ } while (TX_FIFO_LVL(val, sdd) && loops--);
if (loops == 0)
dev_warn(&sdd->pdev->dev, "Timed out flushing TX FIFO\n");
loops = msecs_to_loops(1);
do {
val = readl(regs + S3C64XX_SPI_STATUS);
- if (RX_FIFO_LVL(val, sci))
+ if (RX_FIFO_LVL(val, sdd))
readl(regs + S3C64XX_SPI_RX_DATA);
else
break;
info.direction = sdd->rx_dma.direction;
info.fifo = sdd->sfr_start + S3C64XX_SPI_RX_DATA;
+ info.dt_dmach_prop = sdd->rx_dma.dma_prop;
sdd->rx_dma.ch = sdd->ops->request(sdd->rx_dma.dmach, &info);
info.direction = sdd->tx_dma.direction;
info.fifo = sdd->sfr_start + S3C64XX_SPI_TX_DATA;
+ info.dt_dmach_prop = sdd->tx_dma.dma_prop;
sdd->tx_dma.ch = sdd->ops->request(sdd->tx_dma.dmach, &info);
return 1;
struct spi_device *spi,
struct spi_transfer *xfer, int dma_mode)
{
- struct s3c64xx_spi_info *sci = sdd->cntrlr_info;
void __iomem *regs = sdd->regs;
u32 modecfg, chcfg;
if (xfer->rx_buf != NULL) {
sdd->state |= RXBUSY;
- if (sci->high_speed && sdd->cur_speed >= 30000000UL
+ if (sdd->port_conf->high_speed && sdd->cur_speed >= 30000000UL
&& !(sdd->cur_mode & SPI_CPHA))
chcfg |= S3C64XX_SPI_CH_HS_EN;
if (sdd->tgl_spi != spi) { /* if last mssg on diff device */
/* Deselect the last toggled device */
cs = sdd->tgl_spi->controller_data;
- cs->set_level(cs->line,
- spi->mode & SPI_CS_HIGH ? 0 : 1);
+ gpio_set_value(cs->line,
+ spi->mode & SPI_CS_HIGH ? 0 : 1);
}
sdd->tgl_spi = NULL;
}
cs = spi->controller_data;
- cs->set_level(cs->line, spi->mode & SPI_CS_HIGH ? 1 : 0);
+ gpio_set_value(cs->line, spi->mode & SPI_CS_HIGH ? 1 : 0);
}
static int wait_for_xfer(struct s3c64xx_spi_driver_data *sdd,
struct spi_transfer *xfer, int dma_mode)
{
- struct s3c64xx_spi_info *sci = sdd->cntrlr_info;
void __iomem *regs = sdd->regs;
unsigned long val;
int ms;
val = msecs_to_loops(ms);
do {
status = readl(regs + S3C64XX_SPI_STATUS);
- } while (RX_FIFO_LVL(status, sci) < xfer->len && --val);
+ } while (RX_FIFO_LVL(status, sdd) < xfer->len && --val);
}
if (!val)
if (xfer->rx_buf == NULL) {
val = msecs_to_loops(10);
status = readl(regs + S3C64XX_SPI_STATUS);
- while ((TX_FIFO_LVL(status, sci)
- || !S3C64XX_SPI_ST_TX_DONE(status, sci))
+ while ((TX_FIFO_LVL(status, sdd)
+ || !S3C64XX_SPI_ST_TX_DONE(status, sdd))
&& --val) {
cpu_relax();
status = readl(regs + S3C64XX_SPI_STATUS);
if (sdd->tgl_spi == spi)
sdd->tgl_spi = NULL;
- cs->set_level(cs->line, spi->mode & SPI_CS_HIGH ? 0 : 1);
+ gpio_set_value(cs->line, spi->mode & SPI_CS_HIGH ? 0 : 1);
}
static void s3c64xx_spi_config(struct s3c64xx_spi_driver_data *sdd)
{
- struct s3c64xx_spi_info *sci = sdd->cntrlr_info;
void __iomem *regs = sdd->regs;
u32 val;
/* Disable Clock */
- if (sci->clk_from_cmu) {
+ if (sdd->port_conf->clk_from_cmu) {
clk_disable(sdd->src_clk);
} else {
val = readl(regs + S3C64XX_SPI_CLK_CFG);
writel(val, regs + S3C64XX_SPI_MODE_CFG);
- if (sci->clk_from_cmu) {
+ if (sdd->port_conf->clk_from_cmu) {
/* Configure Clock */
/* There is half-multiplier before the SPI */
clk_set_rate(sdd->src_clk, sdd->cur_speed * 2);
static int s3c64xx_spi_map_mssg(struct s3c64xx_spi_driver_data *sdd,
struct spi_message *msg)
{
- struct s3c64xx_spi_info *sci = sdd->cntrlr_info;
struct device *dev = &sdd->pdev->dev;
struct spi_transfer *xfer;
/* Map until end or first fail */
list_for_each_entry(xfer, &msg->transfers, transfer_list) {
- if (xfer->len <= ((sci->fifo_lvl_mask >> 1) + 1))
+ if (xfer->len <= ((FIFO_LVL_MASK(sdd) >> 1) + 1))
continue;
if (xfer->tx_buf != NULL) {
static void s3c64xx_spi_unmap_mssg(struct s3c64xx_spi_driver_data *sdd,
struct spi_message *msg)
{
- struct s3c64xx_spi_info *sci = sdd->cntrlr_info;
struct device *dev = &sdd->pdev->dev;
struct spi_transfer *xfer;
list_for_each_entry(xfer, &msg->transfers, transfer_list) {
- if (xfer->len <= ((sci->fifo_lvl_mask >> 1) + 1))
+ if (xfer->len <= ((FIFO_LVL_MASK(sdd) >> 1) + 1))
continue;
if (xfer->rx_buf != NULL
struct spi_message *msg)
{
struct s3c64xx_spi_driver_data *sdd = spi_master_get_devdata(master);
- struct s3c64xx_spi_info *sci = sdd->cntrlr_info;
struct spi_device *spi = msg->spi;
struct s3c64xx_spi_csinfo *cs = spi->controller_data;
struct spi_transfer *xfer;
}
/* Polling method for xfers not bigger than FIFO capacity */
- if (xfer->len <= ((sci->fifo_lvl_mask >> 1) + 1))
+ if (xfer->len <= ((FIFO_LVL_MASK(sdd) >> 1) + 1))
use_dma = 0;
else
use_dma = 1;
return 0;
}
+static struct s3c64xx_spi_csinfo *s3c64xx_get_slave_ctrldata(
+ struct s3c64xx_spi_driver_data *sdd,
+ struct spi_device *spi)
+{
+ struct s3c64xx_spi_csinfo *cs;
+ struct device_node *slave_np, *data_np;
+ u32 fb_delay = 0;
+
+ slave_np = spi->dev.of_node;
+ if (!slave_np) {
+ dev_err(&spi->dev, "device node not found\n");
+ return ERR_PTR(-EINVAL);
+ }
+
+ for_each_child_of_node(slave_np, data_np)
+ if (!strcmp(data_np->name, "controller-data"))
+ break;
+ if (!data_np) {
+ dev_err(&spi->dev, "child node 'controller-data' not found\n");
+ return ERR_PTR(-EINVAL);
+ }
+
+ cs = devm_kzalloc(&sdd->pdev->dev, sizeof(*cs), GFP_KERNEL);
+ if (!cs) {
+ dev_err(&spi->dev, "could not allocate memory for controller"
+ " data\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ cs->line = of_get_named_gpio(data_np, "cs-gpio", 0);
+ if (!gpio_is_valid(cs->line)) {
+ dev_err(&spi->dev, "chip select gpio is invalid\n");
+ return ERR_PTR(-EINVAL);
+ }
+ if (devm_gpio_request(&sdd->pdev->dev, cs->line, "spi-cs")) {
+ dev_err(&spi->dev, "gpio [%d] request failed\n", cs->line);
+ return ERR_PTR(-EBUSY);
+ }
+
+ of_property_read_u32(data_np, "samsung,spi-feedback-delay", &fb_delay);
+ cs->fb_delay = fb_delay;
+ return cs;
+}
+
/*
* Here we only check the validity of requested configuration
* and save the configuration in a local data-structure.
unsigned long flags;
int err = 0;
- if (cs == NULL || cs->set_level == NULL) {
+ sdd = spi_master_get_devdata(spi->master);
+ if (!cs && spi->dev.of_node) {
+ cs = s3c64xx_get_slave_ctrldata(sdd, spi);
+ spi->controller_data = cs;
+ }
+
+ if (IS_ERR_OR_NULL(cs)) {
dev_err(&spi->dev, "No CS for SPI(%d)\n", spi->chip_select);
return -ENODEV;
}
- sdd = spi_master_get_devdata(spi->master);
sci = sdd->cntrlr_info;
spin_lock_irqsave(&sdd->lock, flags);
pm_runtime_get_sync(&sdd->pdev->dev);
/* Check if we can provide the requested rate */
- if (!sci->clk_from_cmu) {
+ if (!sdd->port_conf->clk_from_cmu) {
u32 psr, speed;
/* Max possible */
/* Disable Interrupts - we use Polling if not DMA mode */
writel(0, regs + S3C64XX_SPI_INT_EN);
- if (!sci->clk_from_cmu)
+ if (!sdd->port_conf->clk_from_cmu)
writel(sci->src_clk_nr << S3C64XX_SPI_CLKSEL_SRCSHFT,
regs + S3C64XX_SPI_CLK_CFG);
writel(0, regs + S3C64XX_SPI_MODE_CFG);
flush_fifo(sdd);
}
-static int __init s3c64xx_spi_probe(struct platform_device *pdev)
+static int __devinit s3c64xx_spi_get_dmares(
+ struct s3c64xx_spi_driver_data *sdd, bool tx)
+{
+ struct platform_device *pdev = sdd->pdev;
+ struct s3c64xx_spi_dma_data *dma_data;
+ struct property *prop;
+ struct resource *res;
+ char prop_name[15], *chan_str;
+
+ if (tx) {
+ dma_data = &sdd->tx_dma;
+ dma_data->direction = DMA_TO_DEVICE;
+ chan_str = "tx";
+ } else {
+ dma_data = &sdd->rx_dma;
+ dma_data->direction = DMA_FROM_DEVICE;
+ chan_str = "rx";
+ }
+
+ if (!sdd->pdev->dev.of_node) {
+ res = platform_get_resource(pdev, IORESOURCE_DMA, tx ? 0 : 1);
+ if (!res) {
+ dev_err(&pdev->dev, "Unable to get SPI-%s dma "
+ "resource\n", chan_str);
+ return -ENXIO;
+ }
+ dma_data->dmach = res->start;
+ return 0;
+ }
+
+ sprintf(prop_name, "%s-dma-channel", chan_str);
+ prop = of_find_property(pdev->dev.of_node, prop_name, NULL);
+ if (!prop) {
+ dev_err(&pdev->dev, "%s dma channel property not specified\n",
+ chan_str);
+ return -ENXIO;
+ }
+
+ dma_data->dmach = DMACH_DT_PROP;
+ dma_data->dma_prop = prop;
+ return 0;
+}
+
+#ifdef CONFIG_OF
+static int s3c64xx_spi_parse_dt_gpio(struct s3c64xx_spi_driver_data *sdd)
+{
+ struct device *dev = &sdd->pdev->dev;
+ int idx, gpio, ret;
+
+ /* find gpios for mosi, miso and clock lines */
+ for (idx = 0; idx < 3; idx++) {
+ gpio = of_get_gpio(dev->of_node, idx);
+ if (!gpio_is_valid(gpio)) {
+ dev_err(dev, "invalid gpio[%d]: %d\n", idx, gpio);
+ goto free_gpio;
+ }
+
+ ret = gpio_request(gpio, "spi-bus");
+ if (ret) {
+ dev_err(dev, "gpio [%d] request failed\n", gpio);
+ goto free_gpio;
+ }
+ }
+ return 0;
+
+free_gpio:
+ while (--idx >= 0)
+ gpio_free(sdd->gpios[idx]);
+ return -EINVAL;
+}
+
+static void s3c64xx_spi_dt_gpio_free(struct s3c64xx_spi_driver_data *sdd)
+{
+ unsigned int idx;
+ for (idx = 0; idx < 3; idx++)
+ gpio_free(sdd->gpios[idx]);
+}
+
+static struct __devinit s3c64xx_spi_info * s3c64xx_spi_parse_dt(
+ struct device *dev)
{
- struct resource *mem_res, *dmatx_res, *dmarx_res;
- struct s3c64xx_spi_driver_data *sdd;
struct s3c64xx_spi_info *sci;
- struct spi_master *master;
- int ret, irq;
- char clk_name[16];
+ u32 temp;
- if (pdev->id < 0) {
- dev_err(&pdev->dev,
- "Invalid platform device id-%d\n", pdev->id);
- return -ENODEV;
+ sci = devm_kzalloc(dev, sizeof(*sci), GFP_KERNEL);
+ if (!sci) {
+ dev_err(dev, "memory allocation for spi_info failed\n");
+ return ERR_PTR(-ENOMEM);
}
- if (pdev->dev.platform_data == NULL) {
- dev_err(&pdev->dev, "platform_data missing!\n");
- return -ENODEV;
+ if (of_property_read_u32(dev->of_node, "samsung,spi-src-clk", &temp)) {
+ dev_warn(dev, "spi bus clock parent not specified, using "
+ "clock at index 0 as parent\n");
+ sci->src_clk_nr = 0;
+ } else {
+ sci->src_clk_nr = temp;
+ }
+
+ if (of_property_read_u32(dev->of_node, "num-cs", &temp)) {
+ dev_warn(dev, "number of chip select lines not specified, "
+ "assuming 1 chip select line\n");
+ sci->num_cs = 1;
+ } else {
+ sci->num_cs = temp;
}
- sci = pdev->dev.platform_data;
+ return sci;
+}
+#else
+static struct s3c64xx_spi_info *s3c64xx_spi_parse_dt(struct device *dev)
+{
+ return dev->platform_data;
+}
- /* Check for availability of necessary resource */
+static int s3c64xx_spi_parse_dt_gpio(struct s3c64xx_spi_driver_data *sdd)
+{
+ return -EINVAL;
+}
- dmatx_res = platform_get_resource(pdev, IORESOURCE_DMA, 0);
- if (dmatx_res == NULL) {
- dev_err(&pdev->dev, "Unable to get SPI-Tx dma resource\n");
- return -ENXIO;
+static void s3c64xx_spi_dt_gpio_free(struct s3c64xx_spi_driver_data *sdd)
+{
+}
+#endif
+
+static const struct of_device_id s3c64xx_spi_dt_match[];
+
+static inline struct s3c64xx_spi_port_config *s3c64xx_spi_get_port_config(
+ struct platform_device *pdev)
+{
+#ifdef CONFIG_OF
+ if (pdev->dev.of_node) {
+ const struct of_device_id *match;
+ match = of_match_node(s3c64xx_spi_dt_match, pdev->dev.of_node);
+ return (struct s3c64xx_spi_port_config *)match->data;
}
+#endif
+ return (struct s3c64xx_spi_port_config *)
+ platform_get_device_id(pdev)->driver_data;
+}
- dmarx_res = platform_get_resource(pdev, IORESOURCE_DMA, 1);
- if (dmarx_res == NULL) {
- dev_err(&pdev->dev, "Unable to get SPI-Rx dma resource\n");
- return -ENXIO;
+static int __init s3c64xx_spi_probe(struct platform_device *pdev)
+{
+ struct resource *mem_res;
+ struct s3c64xx_spi_driver_data *sdd;
+ struct s3c64xx_spi_info *sci = pdev->dev.platform_data;
+ struct spi_master *master;
+ int ret, irq;
+ char clk_name[16];
+
+ if (!sci && pdev->dev.of_node) {
+ sci = s3c64xx_spi_parse_dt(&pdev->dev);
+ if (IS_ERR(sci))
+ return PTR_ERR(sci);
+ }
+
+ if (!sci) {
+ dev_err(&pdev->dev, "platform_data missing!\n");
+ return -ENODEV;
}
mem_res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
platform_set_drvdata(pdev, master);
sdd = spi_master_get_devdata(master);
+ sdd->port_conf = s3c64xx_spi_get_port_config(pdev);
+
sdd->master = master;
sdd->cntrlr_info = sci;
sdd->pdev = pdev;
sdd->sfr_start = mem_res->start;
- sdd->tx_dma.dmach = dmatx_res->start;
- sdd->tx_dma.direction = DMA_MEM_TO_DEV;
- sdd->rx_dma.dmach = dmarx_res->start;
- sdd->rx_dma.direction = DMA_DEV_TO_MEM;
+ if (pdev->dev.of_node) {
+ ret = of_alias_get_id(pdev->dev.of_node, "spi");
+ if (ret < 0) {
+ dev_err(&pdev->dev, "failed to get alias id, "
+ "errno %d\n", ret);
+ goto err0;
+ }
+ sdd->port_id = ret;
+ } else {
+ sdd->port_id = pdev->id;
+ }
sdd->cur_bpw = 8;
- master->bus_num = pdev->id;
+ ret = s3c64xx_spi_get_dmares(sdd, true);
+ if (ret)
+ goto err0;
+
+ ret = s3c64xx_spi_get_dmares(sdd, false);
+ if (ret)
+ goto err0;
+
+ master->dev.of_node = pdev->dev.of_node;
+ master->bus_num = sdd->port_id;
master->setup = s3c64xx_spi_setup;
master->prepare_transfer_hardware = s3c64xx_spi_prepare_transfer;
master->transfer_one_message = s3c64xx_spi_transfer_one_message;
goto err1;
}
- if (sci->cfg_gpio == NULL || sci->cfg_gpio(pdev)) {
+ if (!sci->cfg_gpio && pdev->dev.of_node) {
+ if (s3c64xx_spi_parse_dt_gpio(sdd))
+ return -EBUSY;
+ } else if (sci->cfg_gpio == NULL || sci->cfg_gpio()) {
dev_err(&pdev->dev, "Unable to config gpio\n");
ret = -EBUSY;
goto err2;
}
/* Setup Deufult Mode */
- s3c64xx_spi_hwinit(sdd, pdev->id);
+ s3c64xx_spi_hwinit(sdd, sdd->port_id);
spin_lock_init(&sdd->lock);
init_completion(&sdd->xfer_completion);
dev_dbg(&pdev->dev, "Samsung SoC SPI Driver loaded for Bus SPI-%d "
"with %d Slaves attached\n",
- pdev->id, master->num_chipselect);
+ sdd->port_id, master->num_chipselect);
dev_dbg(&pdev->dev, "\tIOmem=[0x%x-0x%x]\tDMA=[Rx-%d, Tx-%d]\n",
mem_res->end, mem_res->start,
sdd->rx_dma.dmach, sdd->tx_dma.dmach);
err4:
clk_put(sdd->clk);
err3:
+ if (!sdd->cntrlr_info->cfg_gpio && pdev->dev.of_node)
+ s3c64xx_spi_dt_gpio_free(sdd);
err2:
iounmap((void *) sdd->regs);
err1:
clk_disable(sdd->clk);
clk_put(sdd->clk);
+ if (!sdd->cntrlr_info->cfg_gpio && pdev->dev.of_node)
+ s3c64xx_spi_dt_gpio_free(sdd);
+
iounmap((void *) sdd->regs);
mem_res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
clk_disable(sdd->src_clk);
clk_disable(sdd->clk);
+ if (!sdd->cntrlr_info->cfg_gpio && dev->of_node)
+ s3c64xx_spi_dt_gpio_free(sdd);
+
sdd->cur_speed = 0; /* Output Clock is stopped */
return 0;
static int s3c64xx_spi_resume(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
struct spi_master *master = spi_master_get(dev_get_drvdata(dev));
struct s3c64xx_spi_driver_data *sdd = spi_master_get_devdata(master);
struct s3c64xx_spi_info *sci = sdd->cntrlr_info;
- sci->cfg_gpio(pdev);
+ if (!sci->cfg_gpio && dev->of_node)
+ s3c64xx_spi_parse_dt_gpio(sdd);
+ else
+ sci->cfg_gpio();
+
/* Enable the clock */
clk_enable(sdd->src_clk);
clk_enable(sdd->clk);
- s3c64xx_spi_hwinit(sdd, pdev->id);
+ s3c64xx_spi_hwinit(sdd, sdd->port_id);
spi_master_resume(master);
s3c64xx_spi_runtime_resume, NULL)
};
+struct s3c64xx_spi_port_config s3c2443_spi_port_config = {
+ .fifo_lvl_mask = { 0x7f },
+ .rx_lvl_offset = 13,
+ .tx_st_done = 21,
+ .high_speed = true,
+};
+
+struct s3c64xx_spi_port_config s3c6410_spi_port_config = {
+ .fifo_lvl_mask = { 0x7f, 0x7F },
+ .rx_lvl_offset = 13,
+ .tx_st_done = 21,
+};
+
+struct s3c64xx_spi_port_config s5p64x0_spi_port_config = {
+ .fifo_lvl_mask = { 0x1ff, 0x7F },
+ .rx_lvl_offset = 15,
+ .tx_st_done = 25,
+};
+
+struct s3c64xx_spi_port_config s5pc100_spi_port_config = {
+ .fifo_lvl_mask = { 0x7f, 0x7F },
+ .rx_lvl_offset = 13,
+ .tx_st_done = 21,
+ .high_speed = true,
+};
+
+struct s3c64xx_spi_port_config s5pv210_spi_port_config = {
+ .fifo_lvl_mask = { 0x1ff, 0x7F },
+ .rx_lvl_offset = 15,
+ .tx_st_done = 25,
+ .high_speed = true,
+};
+
+struct s3c64xx_spi_port_config exynos4_spi_port_config = {
+ .fifo_lvl_mask = { 0x1ff, 0x7F, 0x7F },
+ .rx_lvl_offset = 15,
+ .tx_st_done = 25,
+ .high_speed = true,
+ .clk_from_cmu = true,
+};
+
+static struct platform_device_id s3c64xx_spi_driver_ids[] = {
+ {
+ .name = "s3c2443-spi",
+ .driver_data = (kernel_ulong_t)&s3c2443_spi_port_config,
+ }, {
+ .name = "s3c6410-spi",
+ .driver_data = (kernel_ulong_t)&s3c6410_spi_port_config,
+ }, {
+ .name = "s5p64x0-spi",
+ .driver_data = (kernel_ulong_t)&s5p64x0_spi_port_config,
+ }, {
+ .name = "s5pc100-spi",
+ .driver_data = (kernel_ulong_t)&s5pc100_spi_port_config,
+ }, {
+ .name = "s5pv210-spi",
+ .driver_data = (kernel_ulong_t)&s5pv210_spi_port_config,
+ }, {
+ .name = "exynos4210-spi",
+ .driver_data = (kernel_ulong_t)&exynos4_spi_port_config,
+ },
+ { },
+};
+
+#ifdef CONFIG_OF
+static const struct of_device_id s3c64xx_spi_dt_match[] = {
+ { .compatible = "samsung,exynos4210-spi",
+ .data = (void *)&exynos4_spi_port_config,
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, s3c64xx_spi_dt_match);
+#endif /* CONFIG_OF */
+
static struct platform_driver s3c64xx_spi_driver = {
.driver = {
.name = "s3c64xx-spi",
.owner = THIS_MODULE,
.pm = &s3c64xx_spi_pm,
+ .of_match_table = of_match_ptr(s3c64xx_spi_dt_match),
},
.remove = s3c64xx_spi_remove,
+ .id_table = s3c64xx_spi_driver_ids,
};
MODULE_ALIAS("platform:s3c64xx-spi");
config FB_S3C
tristate "Samsung S3C framebuffer support"
- depends on FB && (S3C_DEV_FB || S5P_DEV_FIMD0)
+ depends on FB
select FB_CFB_FILLRECT
select FB_CFB_COPYAREA
select FB_CFB_IMAGEBLIT
Currently the support is only for the S3C6400 and S3C6410 SoCs.
+config FB_EXYNOS_FIMD_V8
+ bool "register extensions for FIMD version 8"
+ depends on ARCH_EXYNOS5
+ ---help---
+ This uses register extensions for FIMD version 8
config FB_S3C_DEBUG_REGWRITE
bool "Debug register writes"
depends on FB_S3C
Turn on debugging messages. Note that you can set/unset at run time
through sysfs
+config FB_MIPI_DSIM
+ bool "Samsung MIPI DSIM"
+ depends on FB_S3C || DRM_EXYNOS_FIMD
+ default n
+ ---help---
+ This enables support for Samsung MIPI DSIM feature
+
config FB_NUC900
bool "NUC900 LCD framebuffer support"
depends on FB && ARCH_W90X900
obj-$(CONFIG_FB_S1D13XXX) += s1d13xxxfb.o
obj-$(CONFIG_FB_SH7760) += sh7760fb.o
obj-$(CONFIG_FB_IMX) += imxfb.o
+obj-$(CONFIG_FB_MIPI_DSIM) += s5p_mipi_dsi.o s5p_mipi_dsi_lowlevel.o
obj-$(CONFIG_FB_S3C) += s3c-fb.o
obj-$(CONFIG_FB_S3C2410) += s3c2410fb.o
obj-$(CONFIG_FB_FSL_DIU) += fsl-diu-fb.o
If you have an S6E63M0 LCD Panel, say Y to enable its
LCD control driver.
+config LCD_MIPI_TC358764
+ tristate "1280 X 800 TC358764 AMOLED MIPI LCD Driver"
+ depends on BACKLIGHT_CLASS_DEVICE
+ default n
+ help
+ If you have an TC358764 MIPI LCD Panel, say Y to enable its
+ LCD control driver.
+
config LCD_LD9040
tristate "LD9040 AMOLED LCD Driver"
depends on SPI && BACKLIGHT_CLASS_DEVICE
obj-$(CONFIG_LCD_TDO24M) += tdo24m.o
obj-$(CONFIG_LCD_TOSA) += tosa_lcd.o
obj-$(CONFIG_LCD_S6E63M0) += s6e63m0.o
+obj-$(CONFIG_LCD_MIPI_TC358764) += tc358764_mipi_lcd.o
obj-$(CONFIG_LCD_LD9040) += ld9040.o
obj-$(CONFIG_LCD_AMS369FG06) += ams369fg06.o
--- /dev/null
+/* linux/drivers/video/backlight/tc358764_mipi_lcd.c
+ *
+ *
+ * Copyright (c) 2011 Samsung Electronics
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ *
+*/
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <linux/mutex.h>
+#include <linux/wait.h>
+#include <linux/ctype.h>
+#include <linux/io.h>
+#include <linux/delay.h>
+#include <linux/irq.h>
+#include <linux/interrupt.h>
+#include <linux/gpio.h>
+#include <linux/workqueue.h>
+#include <linux/backlight.h>
+#include <linux/lcd.h>
+
+#include <video/mipi_display.h>
+
+#include <plat/gpio-cfg.h>
+#include <plat/regs-mipidsim.h>
+
+#include <plat/dsim.h>
+#include <plat/mipi_dsi.h>
+
+static int init_lcd(struct mipi_dsim_device *dsim)
+{
+ unsigned char initcode_013c[6] = {0x3c, 0x01, 0x03, 0x00, 0x02, 0x00};
+ unsigned char initcode_0114[6] = {0x14, 0x01, 0x02, 0x00, 0x00, 0x00};
+ unsigned char initcode_0164[6] = {0x64, 0x01, 0x05, 0x00, 0x00, 0x00};
+ unsigned char initcode_0168[6] = {0x68, 0x01, 0x05, 0x00, 0x00, 0x00};
+ unsigned char initcode_016c[6] = {0x6c, 0x01, 0x05, 0x00, 0x00, 0x00};
+ unsigned char initcode_0170[6] = {0x70, 0x01, 0x05, 0x00, 0x00, 0x00};
+ unsigned char initcode_0134[6] = {0x34, 0x01, 0x1f, 0x00, 0x00, 0x00};
+ unsigned char initcode_0210[6] = {0x10, 0x02, 0x1f, 0x00, 0x00, 0x00};
+ unsigned char initcode_0104[6] = {0x04, 0x01, 0x01, 0x00, 0x00, 0x00};
+ unsigned char initcode_0204[6] = {0x04, 0x02, 0x01, 0x00, 0x00, 0x00};
+ unsigned char initcode_0450[6] = {0x50, 0x04, 0x20, 0x01, 0xfa, 0x00};
+ unsigned char initcode_0454[6] = {0x54, 0x04, 0x20, 0x00, 0x50, 0x00};
+ unsigned char initcode_0458[6] = {0x58, 0x04, 0x00, 0x05, 0x30, 0x00};
+ unsigned char initcode_045c[6] = {0x5c, 0x04, 0x05, 0x00, 0x0a, 0x00};
+ unsigned char initcode_0460[6] = {0x60, 0x04, 0x20, 0x03, 0x0a, 0x00};
+ unsigned char initcode_0464[6] = {0x64, 0x04, 0x01, 0x00, 0x00, 0x00};
+ unsigned char initcode_04a0_1[6] = {0xa0, 0x04, 0x06, 0x80, 0x44, 0x00};
+ unsigned char initcode_04a0_2[6] = {0xa0, 0x04, 0x06, 0x80, 0x04, 0x00};
+ unsigned char initcode_0504[6] = {0x04, 0x05, 0x04, 0x00, 0x00, 0x00};
+ unsigned char initcode_049c[6] = {0x9c, 0x04, 0x0d, 0x00, 0x00, 0x00};
+
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_013c, sizeof(initcode_013c)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0114, sizeof(initcode_0114)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0164, sizeof(initcode_0164)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0168, sizeof(initcode_0168)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_016c, sizeof(initcode_016c)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0170, sizeof(initcode_0170)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0134, sizeof(initcode_0134)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0210, sizeof(initcode_0210)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0104, sizeof(initcode_0104)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0204, sizeof(initcode_0204)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0450, sizeof(initcode_0450)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0454, sizeof(initcode_0454)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0458, sizeof(initcode_0458)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_045c, sizeof(initcode_045c)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0460, sizeof(initcode_0460)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0464, sizeof(initcode_0464)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_04a0_1, sizeof(initcode_04a0_1)) == -1)
+ return 0;
+ mdelay(12);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_04a0_2, sizeof(initcode_04a0_2)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_0504, sizeof(initcode_0504)) == -1)
+ return 0;
+ mdelay(6);
+ if (s5p_mipi_dsi_wr_data(dsim, MIPI_DSI_GENERIC_LONG_WRITE,
+ (unsigned int) initcode_049c, sizeof(initcode_049c)) == -1)
+ return 0;
+ mdelay(800);
+
+ return 1;
+}
+
+void tc358764_mipi_lcd_off(struct mipi_dsim_device *dsim)
+{
+ mdelay(1);
+}
+
+static int tc358764_mipi_lcd_bl_update_status(struct backlight_device *bd)
+{
+ return 0;
+}
+
+static const struct backlight_ops tc358764_mipi_lcd_bl_ops = {
+ .update_status = tc358764_mipi_lcd_bl_update_status,
+};
+
+static int tc358764_mipi_lcd_probe(struct mipi_dsim_device *dsim)
+{
+ struct mipi_dsim_device *dsim_drv;
+ struct backlight_device *bd = NULL;
+ struct backlight_properties props;
+
+ dsim_drv = kzalloc(sizeof(struct mipi_dsim_device), GFP_KERNEL);
+ if (!dsim_drv)
+ return -ENOMEM;
+
+ dsim_drv = (struct mipi_dsim_device *) dsim;
+
+ props.max_brightness = 1;
+ props.type = BACKLIGHT_PLATFORM;
+
+ bd = backlight_device_register("pwm-backlight",
+ dsim_drv->dev, dsim_drv, &tc358764_mipi_lcd_bl_ops, &props);
+
+ return 0;
+}
+
+static int tc358764_mipi_lcd_suspend(struct mipi_dsim_device *dsim)
+{
+ tc358764_mipi_lcd_off(dsim);
+ return 0;
+}
+
+static int tc358764_mipi_lcd_displayon(struct mipi_dsim_device *dsim)
+{
+ return init_lcd(dsim);
+}
+
+static int tc358764_mipi_lcd_resume(struct mipi_dsim_device *dsim)
+{
+ return init_lcd(dsim);
+}
+
+struct mipi_dsim_lcd_driver tc358764_mipi_lcd_driver = {
+ .probe = tc358764_mipi_lcd_probe,
+ .suspend = tc358764_mipi_lcd_suspend,
+ .displayon = tc358764_mipi_lcd_displayon,
+ .resume = tc358764_mipi_lcd_resume,
+};
#include <linux/io.h>
#include <linux/interrupt.h>
#include <linux/delay.h>
+#include <linux/of.h>
#include <video/exynos_dp.h>
/* SW defined function Normal operation */
exynos_dp_enable_sw_function(dp);
- exynos_dp_config_interrupt(dp);
exynos_dp_init_analog_func(dp);
exynos_dp_init_hpd(dp);
struct exynos_dp_device *dp,
int lane)
{
- u32 reg;
+ u32 reg = 0;
switch (lane) {
case 0:
int lane_count;
u8 buf[5];
- u8 *adjust_request;
- u8 voltage_swing;
+ u8 adjust_request[2];
+ u8 voltage_swing = 0;
u8 pre_emphasis;
u8 training_lane;
/* set training pattern 2 for EQ */
exynos_dp_set_training_pattern(dp, TRAINING_PTN2);
- adjust_request = link_status + (DPCD_ADDR_ADJUST_REQUEST_LANE0_1
- - DPCD_ADDR_LANE0_1_STATUS);
+ adjust_request[0] = link_status[4];
+ adjust_request[1] = link_status[5];
exynos_dp_get_adjust_train(dp, adjust_request);
u8 buf[5];
u32 reg;
- u8 *adjust_request;
+ u8 adjust_request[2];
udelay(400);
lane_count = dp->link_train.lane_count;
if (exynos_dp_clock_recovery_ok(link_status, lane_count) == 0) {
- adjust_request = link_status + (DPCD_ADDR_ADJUST_REQUEST_LANE0_1
- - DPCD_ADDR_LANE0_1_STATUS);
+ adjust_request[0] = link_status[4];
+ adjust_request[1] = link_status[5];
if (exynos_dp_channel_eq_ok(link_status, lane_count) == 0) {
/* traing pattern Set to Normal */
return retval;
}
+static int exynos_dp_set_hw_link_train(struct exynos_dp_device *dp,
+ u32 max_lane,
+ u32 max_rate)
+{
+ u32 status;
+ int lane;
+
+ exynos_dp_stop_video(dp);
+
+ if (exynos_dp_get_pll_lock_status(dp) == PLL_UNLOCKED) {
+ dev_err(dp->dev, "PLL is not locked yet.\n");
+ return -EINVAL;
+ }
+
+ exynos_dp_reset_macro(dp);
+
+ /* Set TX pre-emphasis to minimum */
+ for (lane = 0; lane < max_lane; lane++)
+ exynos_dp_set_lane_lane_pre_emphasis(dp,
+ PRE_EMPHASIS_LEVEL_0, lane);
+
+ /* All DP analog module power up */
+ exynos_dp_set_analog_power_down(dp, POWER_ALL, 0);
+
+ /* Initialize by reading RX's DPCD */
+ exynos_dp_get_max_rx_bandwidth(dp, &dp->link_train.link_rate);
+ exynos_dp_get_max_rx_lane_count(dp, &dp->link_train.lane_count);
+
+ if ((dp->link_train.link_rate != LINK_RATE_1_62GBPS) &&
+ (dp->link_train.link_rate != LINK_RATE_2_70GBPS)) {
+ dev_err(dp->dev, "Rx Max Link Rate is abnormal :%x !\n",
+ dp->link_train.link_rate);
+ dp->link_train.link_rate = LINK_RATE_1_62GBPS;
+ }
+
+ if (dp->link_train.lane_count == 0) {
+ dev_err(dp->dev, "Rx Max Lane count is abnormal :%x !\n",
+ dp->link_train.lane_count);
+ dp->link_train.lane_count = (u8)LANE_COUNT1;
+ }
+
+ /* Setup TX lane count & rate */
+ if (dp->link_train.lane_count > max_lane)
+ dp->link_train.lane_count = max_lane;
+ if (dp->link_train.link_rate > max_rate)
+ dp->link_train.link_rate = max_rate;
+
+ /* Set link rate and count as you want to establish*/
+ exynos_dp_set_lane_count(dp, dp->video_info->lane_count);
+ exynos_dp_set_link_bandwidth(dp, dp->video_info->link_rate);
+
+ /* Set sink to D0 (Sink Not Ready) mode. */
+ exynos_dp_write_byte_to_dpcd(dp, DPCD_ADDR_SINK_POWER_STATE,
+ DPCD_SET_POWER_STATE_D0);
+
+ /* Enable H/W Link Training */
+ status = exynos_dp_enable_hw_link_training(dp);
+
+ if (status != 0) {
+ dev_err(dp->dev, " H/W link training failure: 0x%x\n", status);
+ return -EINVAL;
+ }
+
+ exynos_dp_get_link_bandwidth(dp, &status);
+ dp->link_train.link_rate = status;
+ dev_dbg(dp->dev, "final bandwidth = %.2x\n",
+ dp->link_train.link_rate);
+
+ exynos_dp_get_lane_count(dp, &status);
+ dp->link_train.lane_count = status;
+ dev_dbg(dp->dev, "final lane count = %.2x\n",
+ dp->link_train.lane_count);
+
+ return 0;
+}
+
static int exynos_dp_set_link_train(struct exynos_dp_device *dp,
u32 count,
u32 bwtype)
goto err_ioremap;
}
+ dp->video_info = pdata->video_info;
+ if (pdata->phy_init)
+ pdata->phy_init();
+
+ exynos_dp_init_dp(dp);
+
ret = request_irq(dp->irq, exynos_dp_irq_handler, 0,
"exynos-dp", dp);
if (ret) {
goto err_ioremap;
}
- dp->video_info = pdata->video_info;
- if (pdata->phy_init)
- pdata->phy_init();
-
- exynos_dp_init_dp(dp);
-
ret = exynos_dp_detect_hpd(dp);
if (ret) {
dev_err(&pdev->dev, "unable to detect hpd\n");
exynos_dp_handle_edid(dp);
- ret = exynos_dp_set_link_train(dp, dp->video_info->lane_count,
- dp->video_info->link_rate);
+ if (pdata->training_type == SW_LINK_TRAINING)
+ ret = exynos_dp_set_link_train(dp, dp->video_info->lane_count,
+ dp->video_info->link_rate);
+ else
+ ret = exynos_dp_set_hw_link_train(dp,
+ dp->video_info->lane_count, dp->video_info->link_rate);
if (ret) {
dev_err(&pdev->dev, "unable to do link train\n");
goto err_irq;
exynos_dp_detect_hpd(dp);
exynos_dp_handle_edid(dp);
- exynos_dp_set_link_train(dp, dp->video_info->lane_count,
- dp->video_info->link_rate);
+ if (pdata->training_type == SW_LINK_TRAINING)
+ exynos_dp_set_link_train(dp, dp->video_info->lane_count,
+ dp->video_info->link_rate);
+ else
+ exynos_dp_set_hw_link_train(dp,
+ dp->video_info->lane_count, dp->video_info->link_rate);
exynos_dp_enable_scramble(dp, 1);
exynos_dp_enable_rx_to_enhanced_mode(dp, 1);
SET_SYSTEM_SLEEP_PM_OPS(exynos_dp_suspend, exynos_dp_resume)
};
+#ifdef CONFIG_OF
+static const struct of_device_id exynos_dp_match[] = {
+ { .compatible = "samsung,exynos5-dp" },
+ {},
+};
+MODULE_DEVICE_TABLE(of, exynos_dp_match);
+#endif
+
static struct platform_driver exynos_dp_driver = {
.probe = exynos_dp_probe,
.remove = __devexit_p(exynos_dp_remove),
.driver = {
- .name = "exynos-dp",
+ .name = "s5p-dp",
.owner = THIS_MODULE,
.pm = &exynos_dp_pm_ops,
+ .of_match_table = of_match_ptr(exynos_dp_match),
},
};
-module_platform_driver(exynos_dp_driver);
+static int __init exynos_dp_init(void)
+{
+ return platform_driver_probe(&exynos_dp_driver, exynos_dp_probe);
+}
+
+static void __exit exynos_dp_exit(void)
+{
+ platform_driver_unregister(&exynos_dp_driver);
+}
+/* TODO: Register as module_platform_driver */
+/* Currently, we make it late_initcall to make */
+/* sure that s3c-fb is probed before DP driver */
+late_initcall(exynos_dp_init);
+module_exit(exynos_dp_exit);
MODULE_AUTHOR("Jingoo Han <jg1.han@samsung.com>");
MODULE_DESCRIPTION("Samsung SoC DP Driver");
void exynos_dp_lane_swap(struct exynos_dp_device *dp, bool enable);
void exynos_dp_init_interrupt(struct exynos_dp_device *dp);
void exynos_dp_reset(struct exynos_dp_device *dp);
-void exynos_dp_config_interrupt(struct exynos_dp_device *dp);
u32 exynos_dp_get_pll_lock_status(struct exynos_dp_device *dp);
void exynos_dp_set_pll_power_down(struct exynos_dp_device *dp, bool enable);
void exynos_dp_set_analog_power_down(struct exynos_dp_device *dp,
void exynos_dp_enable_scrambling(struct exynos_dp_device *dp);
void exynos_dp_disable_scrambling(struct exynos_dp_device *dp);
+u32 exynos_dp_enable_hw_link_training(struct exynos_dp_device *dp);
+
/* I2C EDID Chip ID, Slave Address */
#define I2C_EDID_DEVICE_ADDR 0x50
#define I2C_E_EDID_DEVICE_ADDR 0x30
#include <linux/device.h>
#include <linux/io.h>
#include <linux/delay.h>
+#include <linux/jiffies.h>
#include <video/exynos_dp.h>
#include "exynos_dp_core.h"
#include "exynos_dp_reg.h"
-#define COMMON_INT_MASK_1 (0)
-#define COMMON_INT_MASK_2 (0)
-#define COMMON_INT_MASK_3 (0)
-#define COMMON_INT_MASK_4 (0)
-#define INT_STA_MASK (0)
-
void exynos_dp_enable_video_mute(struct exynos_dp_device *dp, bool enable)
{
u32 reg;
writel(reg, dp->reg_base + EXYNOS_DP_LANE_MAP);
}
+void exynos_dp_init_analog_param(struct exynos_dp_device *dp)
+{
+ /* Set analog parameters for Tx */
+ /* Set power source and terminal resistor values */
+ writel(0x10, dp->reg_base + EXYNOS_DP_ANALOG_CTL_1);
+ writel(0x0C, dp->reg_base + EXYNOS_DP_ANALOG_CTL_2);
+ writel(0x85, dp->reg_base + EXYNOS_DP_ANALOG_CTL_3);
+ writel(0x66, dp->reg_base + EXYNOS_DP_PLL_FILTER_CTL_1);
+ writel(0x0, dp->reg_base + EXYNOS_DP_TX_AMP_TUNING_CTL);
+}
+
void exynos_dp_init_interrupt(struct exynos_dp_device *dp)
{
/* Set interrupt pin assertion polarity as high */
writel(0x00000101, dp->reg_base + EXYNOS_DP_SOC_GENERAL_CTL);
+ exynos_dp_init_analog_param(dp);
exynos_dp_init_interrupt(dp);
}
-void exynos_dp_config_interrupt(struct exynos_dp_device *dp)
-{
- u32 reg;
-
- /* 0: mask, 1: unmask */
- reg = COMMON_INT_MASK_1;
- writel(reg, dp->reg_base + EXYNOS_DP_COMMON_INT_MASK_1);
-
- reg = COMMON_INT_MASK_2;
- writel(reg, dp->reg_base + EXYNOS_DP_COMMON_INT_MASK_2);
-
- reg = COMMON_INT_MASK_3;
- writel(reg, dp->reg_base + EXYNOS_DP_COMMON_INT_MASK_3);
-
- reg = COMMON_INT_MASK_4;
- writel(reg, dp->reg_base + EXYNOS_DP_COMMON_INT_MASK_4);
-
- reg = INT_STA_MASK;
- writel(reg, dp->reg_base + EXYNOS_DP_INT_STA_MASK);
-}
-
u32 exynos_dp_get_pll_lock_status(struct exynos_dp_device *dp)
{
u32 reg;
void exynos_dp_init_analog_func(struct exynos_dp_device *dp)
{
u32 reg;
+ int timeout_loop = 0;
exynos_dp_set_analog_power_down(dp, POWER_ALL, 0);
writel(reg, dp->reg_base + EXYNOS_DP_DEBUG_CTL);
/* Power up PLL */
- if (exynos_dp_get_pll_lock_status(dp) == PLL_UNLOCKED)
+ if (exynos_dp_get_pll_lock_status(dp) == PLL_UNLOCKED) {
exynos_dp_set_pll_power_down(dp, 0);
+ while (exynos_dp_get_pll_lock_status(dp) == PLL_UNLOCKED) {
+ timeout_loop++;
+ if (DP_TIMEOUT_LOOP_COUNT < timeout_loop) {
+ dev_err(dp->dev, "failed to get pll lock status\n");
+ return;
+ }
+ udelay(10);
+ }
+ }
+
/* Enable Serdes FIFO function and Link symbol clock domain module */
reg = readl(dp->reg_base + EXYNOS_DP_FUNC_EN_2);
reg &= ~(SERDES_FIFO_FUNC_EN_N | LS_CLK_DOMAIN_FUNC_EN_N
reg |= SCRAMBLING_DISABLE;
writel(reg, dp->reg_base + EXYNOS_DP_TRAINING_PTN_SET);
}
+
+u32 exynos_dp_enable_hw_link_training(struct exynos_dp_device *dp)
+{
+ u32 reg;
+ unsigned long timeout;
+
+ reg = HW_TRAINING_EN;
+ writel(reg, dp->reg_base + EXYNOS_DP_HW_LINK_TRAINING_CTL);
+
+ /* wait for maximum of 100 msec */
+ timeout = jiffies + msecs_to_jiffies(100);
+ do {
+ reg = readl(dp->reg_base + EXYNOS_DP_HW_LINK_TRAINING_CTL);
+ if (!(reg & HW_TRAINING_EN))
+ return 0;
+ udelay(10);
+ } while (time_before(jiffies, timeout));
+
+ dev_warn(dp->dev, "H/W Link training failed\n");
+ return -ETIMEDOUT;
+}
#define EXYNOS_DP_LANE_MAP 0x35C
+#define EXYNOS_DP_ANALOG_CTL_1 0x370
+#define EXYNOS_DP_ANALOG_CTL_2 0x374
+#define EXYNOS_DP_ANALOG_CTL_3 0x378
+#define EXYNOS_DP_PLL_FILTER_CTL_1 0x37C
+#define EXYNOS_DP_TX_AMP_TUNING_CTL 0x380
+
#define EXYNOS_DP_AUX_HW_RETRY_CTL 0x390
#define EXYNOS_DP_COMMON_INT_STA_1 0x3C4
#define VIDEO_MODE_SLAVE_MODE (0x1 << 0)
#define VIDEO_MODE_MASTER_MODE (0x0 << 0)
+#define EXYNOS_DP_HW_LINK_TRAINING_CTL 0x6A0
+#define HW_TRAINING_EN (0x1<<0)
+
#endif /* _EXYNOS_DP_REG_H */
#include <mach/map.h>
#include <plat/regs-fb-v4.h>
#include <plat/fb.h>
+#include <mach/regs-pmu.h>
/* This driver will export a number of framebuffer interfaces depending
* on the configuration passed in via the platform data. Each fb instance
unsigned int has_prtcon:1;
unsigned int has_shadowcon:1;
unsigned int has_blendcon:1;
+ unsigned int has_alphacon:1;
unsigned int has_clksel:1;
unsigned int has_fixvclk:1;
};
-/**
+/*
* struct s3c_fb_win_variant
* @has_osd_c: Set if has OSD C register.
* @has_osd_d: Set if has OSD D register.
{
struct fb_var_screeninfo *var = &info->var;
struct s3c_fb_win *win = info->par;
+ struct s3c_fb_pd_win *windata = win->windata;
struct s3c_fb *sfb = win->parent;
void __iomem *regs = sfb->regs;
void __iomem *buf = regs;
break;
}
+ if (!win->index) {
+ var->xres_virtual = windata->virtual_x;
+ var->yres_virtual = windata->virtual_y;
+ }
info->fix.line_length = (var->xres_virtual * var->bits_per_pixel) / 8;
info->fix.xpanstep = info->var.xres_virtual > info->var.xres ? 1 : 0;
if (sfb->variant.is_2443)
data |= (1 << 5);
+ data |= VIDCON0_ENVID | VIDCON0_ENVID_F;
writel(data, regs + VIDCON0);
- s3c_fb_enable(sfb, 1);
-
data = VIDTCON0_VBPD(var->upper_margin - 1) |
VIDTCON0_VFPD(var->lower_margin - 1) |
VIDTCON0_VSPW(var->vsync_len - 1);
/* VIDTCON1 */
writel(data, regs + sfb->variant.vidtcon + 4);
- data = VIDTCON2_LINEVAL(var->yres - 1) |
- VIDTCON2_HOZVAL(var->xres - 1) |
- VIDTCON2_LINEVAL_E(var->yres - 1) |
- VIDTCON2_HOZVAL_E(var->xres - 1);
+ data = VIDTCON2_LINEVAL(windata->win_mode.yres - 1) |
+ VIDTCON2_HOZVAL(windata->win_mode.xres - 1) |
+ VIDTCON2_LINEVAL_E(windata->win_mode.yres - 1) |
+ VIDTCON2_HOZVAL_E(windata->win_mode.xres - 1);
writel(data, regs + sfb->variant.vidtcon + 8);
}
/* write 'OSD' registers to control position of framebuffer */
- data = VIDOSDxA_TOPLEFT_X(0) | VIDOSDxA_TOPLEFT_Y(0) |
- VIDOSDxA_TOPLEFT_X_E(0) | VIDOSDxA_TOPLEFT_Y_E(0);
+ data = VIDOSDxA_TOPLEFT_X(0) | VIDOSDxA_TOPLEFT_Y(0);
writel(data, regs + VIDOSD_A(win_no, sfb->variant));
data = VIDOSDxB_BOTRIGHT_X(s3c_fb_align_word(var->bits_per_pixel,
writel(data, sfb->regs + SHADOWCON);
}
- data = WINCONx_ENWIN;
- sfb->enabled |= (1 << win->index);
+ if (win_no == sfb->pdata->default_win) {
+ data = WINCONx_ENWIN;
+ sfb->enabled |= (1 << win->index);
+ } else
+ data = 0;
/* note, since we have to round up the bits-per-pixel, we end up
* relying on the bitfield information for r/g/b/a to work out
/* we're stuck with this until we can do something about overriding
* the power control using the blanking event for a single fb.
*/
- if (index == sfb->pdata->default_win) {
- shadow_protect_win(win, 1);
- s3c_fb_enable(sfb, blank_mode != FB_BLANK_POWERDOWN ? 1 : 0);
- shadow_protect_win(win, 0);
- }
+ shadow_protect_win(win, 1);
+ s3c_fb_enable(sfb, sfb->enabled ? 1 : 0);
+ shadow_protect_win(win, 0);
pm_runtime_put_sync(sfb->dev);
return 0;
}
+struct s3c_fb_user_window {
+ int x;
+ int y;
+};
+
+struct s3c_fb_user_plane_alpha {
+ int channel;
+ unsigned char red;
+ unsigned char green;
+ unsigned char blue;
+};
+
+struct s3c_fb_user_chroma {
+ int enabled;
+ unsigned char red;
+ unsigned char green;
+ unsigned char blue;
+};
+
+struct s3c_fb_user_ion_client {
+ int fd;
+};
+
+int s3c_fb_set_window_position(struct fb_info *info,
+ struct s3c_fb_user_window user_window)
+{
+ struct s3c_fb_win *win = info->par;
+ struct s3c_fb *sfb = win->parent;
+ struct fb_var_screeninfo *var = &info->var;
+ int win_no = win->index;
+ void __iomem *regs = sfb->regs;
+ u32 data;
+
+ shadow_protect_win(win, 1);
+ /* write 'OSD' registers to control position of framebuffer */
+ data = VIDOSDxA_TOPLEFT_X(user_window.x) |
+ VIDOSDxA_TOPLEFT_Y(user_window.y) |
+ VIDOSDxA_TOPLEFT_X_E(user_window.x) |
+ VIDOSDxA_TOPLEFT_Y_E(user_window.y);
+ writel(data, regs + VIDOSD_A(win_no, sfb->variant));
+
+ data = VIDOSDxB_BOTRIGHT_X(s3c_fb_align_word(var->bits_per_pixel,
+ user_window.x + var->xres - 1)) |
+ VIDOSDxB_BOTRIGHT_Y(user_window.y + var->yres - 1) |
+ VIDOSDxB_BOTRIGHT_X_E(s3c_fb_align_word(var->bits_per_pixel,
+ user_window.x + var->xres - 1)) |
+ VIDOSDxB_BOTRIGHT_Y_E(user_window.y + var->yres - 1);
+ writel(data, regs + VIDOSD_B(win_no, sfb->variant));
+
+ shadow_protect_win(win, 0);
+ return 0;
+}
+
+int s3c_fb_set_plane_alpha_blending(struct fb_info *info,
+ struct s3c_fb_user_plane_alpha user_alpha)
+{
+ struct s3c_fb_win *win = info->par;
+ struct s3c_fb *sfb = win->parent;
+ int win_no = win->index;
+ void __iomem *regs = sfb->regs;
+ u32 data;
+
+ u32 alpha_high = 0;
+ u32 alpha_low = 0;
+
+ alpha_high = ((((user_alpha.red & 0xf0) >> 4) << 8) |
+ (((user_alpha.green & 0xf0) >> 4) << 4) |
+ (((user_alpha.blue & 0xf0) >> 4) << 0));
+
+ alpha_low = ((((user_alpha.red & 0xf)) << 16) |
+ (((user_alpha.green & 0xf)) << 8) |
+ (((user_alpha.blue & 0xf)) << 0));
+
+ shadow_protect_win(win, 1);
+
+ data = readl(regs + sfb->variant.wincon + (win_no * 4));
+ data &= ~(WINCON1_BLD_PIX | WINCON1_ALPHA_SEL);
+ data |= WINCON1_BLD_PLANE;
+
+ if (user_alpha.channel == 0)
+ alpha_high = alpha_high << 12;
+ else {
+ data |= WINCON1_ALPHA_SEL;
+ alpha_high = alpha_high << 0;
+ }
+
+ writel(data, regs + sfb->variant.wincon + (win_no * 4));
+ writel(alpha_high, regs + VIDOSD_C(win_no, sfb->variant));
+
+ if (sfb->variant.has_alphacon) {
+ if (user_alpha.channel == 0)
+ writel(alpha_low, regs + VIDW0ALPHA0 + (win_no * 8));
+ else
+ writel(alpha_low, regs + VIDW0ALPHA1 + (win_no * 8));
+ }
+
+ shadow_protect_win(win, 0);
+
+ return 0;
+}
+
+int s3c_fb_set_chroma_key(struct fb_info *info,
+ struct s3c_fb_user_chroma user_chroma)
+{
+ struct s3c_fb_win *win = info->par;
+ struct s3c_fb *sfb = win->parent;
+ int win_no = win->index;
+ void __iomem *regs = sfb->regs;
+ void __iomem *keycon = regs + sfb->variant.keycon;
+
+ u32 data = 0;
+
+ u32 chroma_value;
+
+ chroma_value = (((user_chroma.red & 0xff) << 16) |
+ ((user_chroma.green & 0xff) << 8) |
+ ((user_chroma.blue & 0xff) << 0));
+
+ shadow_protect_win(win, 1);
+
+ if (user_chroma.enabled)
+ data |= WxKEYCON0_KEYEN_F;
+
+ keycon += (win_no-1) * 8;
+ writel(data, keycon + WKEYCON0);
+
+ data = (chroma_value & 0xffffff);
+ writel(data, keycon + WKEYCON1);
+
+ shadow_protect_win(win, 0);
+
+ return 0;
+}
+
static int s3c_fb_ioctl(struct fb_info *info, unsigned int cmd,
unsigned long arg)
{
int ret;
u32 crtc;
+ union {
+ struct s3c_fb_user_window user_window;
+ struct s3c_fb_user_plane_alpha user_alpha;
+ struct s3c_fb_user_chroma user_chroma;
+ struct s3c_fb_user_ion_client user_ion_client;
+ } p;
switch (cmd) {
case FBIO_WAITFORVSYNC:
if (get_user(crtc, (u32 __user *)arg)) {
ret = s3c_fb_wait_for_vsync(sfb, crtc);
break;
+
+ case S3CFB_WIN_POSITION:
+ if (copy_from_user(&p.user_window,
+ (struct s3c_fb_user_window __user *)arg,
+ sizeof(p.user_window))) {
+ ret = -EFAULT;
+ break;
+ }
+
+ if (p.user_window.x < 0)
+ p.user_window.x = 0;
+ if (p.user_window.y < 0)
+ p.user_window.y = 0;
+
+ ret = s3c_fb_set_window_position(info, p.user_window);
+ break;
+
+ case S3CFB_WIN_SET_PLANE_ALPHA:
+ if (copy_from_user(&p.user_alpha,
+ (struct s3c_fb_user_plane_alpha __user *)arg,
+ sizeof(p.user_alpha))) {
+ ret = -EFAULT;
+ break;
+ }
+
+ ret = s3c_fb_set_plane_alpha_blending(info, p.user_alpha);
+ break;
+
+ case S3CFB_WIN_SET_CHROMA:
+ if (copy_from_user(&p.user_chroma,
+ (struct s3c_fb_user_chroma __user *)arg,
+ sizeof(p.user_chroma))) {
+ ret = -EFAULT;
+ break;
+ }
+
+ ret = s3c_fb_set_chroma_key(info, p.user_chroma);
+ break;
+
+ case S3CFB_SET_VSYNC_INT:
+ /* unnecessary, but for compatibility */
+ ret = 0;
+ break;
+
default:
ret = -ENOTTY;
}
fbinfo->var.activate = FB_ACTIVATE_NOW;
fbinfo->var.vmode = FB_VMODE_NONINTERLACED;
fbinfo->var.bits_per_pixel = windata->default_bpp;
+ fbinfo->var.width = windata->width;
+ fbinfo->var.height = windata->height;
fbinfo->fbops = &s3c_fb_ops;
fbinfo->flags = FBINFO_FLAG_DEFAULT;
fbinfo->pseudo_palette = &win->pseudo_palette;
struct s3c_fb_platdata *pd;
struct s3c_fb *sfb;
struct resource *res;
- int win;
+ int i, win, default_win;
int ret = 0;
u32 reg;
+ struct clk *clk_parent;
+ struct clk *sclk;
+
+ pd = pdev->dev.platform_data;
+ if (!pd) {
+ dev_err(dev, "no platform data specified\n");
+ return -EINVAL;
+ }
+
+ /* HACK: This should be added from pdata/device tree */
+ sclk = clk_get(&pdev->dev, "sclk_fimd");
+ if (IS_ERR(sclk))
+ return PTR_ERR(sclk);
+
+ clk_parent = clk_get(NULL, "mout_mpll_user");
+ if (IS_ERR(clk_parent)) {
+ clk_put(sclk);
+ return PTR_ERR(clk_parent);
+ }
+
+ if (clk_set_parent(sclk, clk_parent)) {
+ clk_put(sclk);
+ clk_put(clk_parent);
+ return PTR_ERR(sclk);
+ }
+
+ if (clk_set_rate(sclk, pd->clock_rate)) {
+ clk_put(sclk);
+ clk_put(clk_parent);
+ return PTR_ERR(sclk);
+ }
+
+ clk_put(sclk);
+ clk_put(clk_parent);
platid = platform_get_device_id(pdev);
fbdrv = (struct s3c_fb_driverdata *)platid->driver_data;
return -EINVAL;
}
- pd = pdev->dev.platform_data;
- if (!pd) {
- dev_err(dev, "no platform data specified\n");
- return -EINVAL;
- }
-
sfb = devm_kzalloc(dev, sizeof(struct s3c_fb), GFP_KERNEL);
if (!sfb) {
dev_err(dev, "no memory for framebuffers\n");
writel(reg, sfb->regs + VIDCON1);
}
+ /* disable auto-clock gate mode */
+ writel(REG_CLKGATE_MODE_NON_CLOCK_GATE, sfb->regs + REG_CLKGATE_MODE);
+
/* zero all windows before we do anything */
for (win = 0; win < fbdrv->variant.nr_windows; win++)
}
/* we have the register setup, start allocating framebuffers */
-
- for (win = 0; win < fbdrv->variant.nr_windows; win++) {
+ default_win = sfb->pdata->default_win;
+ for (i = 0; i < fbdrv->variant.nr_windows; i++) {
+ win = i;
+ if (i == 0)
+ win = default_win;
+ if (i == default_win)
+ win = 0;
if (!pd->win[win])
continue;
}
}
+ if (pd->panel_type == DP_LCD)
+ writel(DPCLKCON_ENABLE, sfb->regs + DPCLKCON);
+
platform_set_drvdata(pdev, sfb);
pm_runtime_put_sync(sfb->dev);
return 0;
}
+static inline void s3c_fb_enable_fimd_bypass_disp1(void)
+{
+ u32 reg;
+
+ reg = __raw_readl(EXYNOS5_SYS_DISP1BLK_CFG);
+ reg |= ENABLE_FIMDBYPASS_DISP1;
+ __raw_writel(reg, EXYNOS5_SYS_DISP1BLK_CFG);
+}
+
#ifdef CONFIG_PM_SLEEP
static int s3c_fb_suspend(struct device *dev)
{
{
struct platform_device *pdev = to_platform_device(dev);
struct s3c_fb *sfb = platform_get_drvdata(pdev);
- struct s3c_fb_platdata *pd = sfb->pdata;
+ struct s3c_fb_platdata *pd;
struct s3c_fb_win *win;
int win_no;
u32 reg;
if (!sfb->variant.has_clksel)
clk_enable(sfb->lcd_clk);
- /* setup gpio and output polarity controls */
- pd->setup_gpio();
- writel(pd->vidcon1, sfb->regs + VIDCON1);
+ s3c_fb_enable_fimd_bypass_disp1();
+
+ writel(VIDCON1_INV_VCLK, sfb->regs + VIDCON1);
/* set video clock running at under-run */
if (sfb->variant.has_fixvclk) {
writel(reg, sfb->regs + VIDCON1);
}
+ /* disable auto-clock gate mode */
+ writel(REG_CLKGATE_MODE_NON_CLOCK_GATE, sfb->regs + REG_CLKGATE_MODE);
/* zero all windows before we do anything */
for (win_no = 0; win_no < sfb->variant.nr_windows; win_no++)
s3c_fb_clear_win(sfb, win_no);
s3c_fb_set_par(win->fbinfo);
}
+ pd = pdev->dev.platform_data;
+
+ if (pd->panel_type == DP_LCD)
+ writel(DPCLKCON_ENABLE, sfb->regs + DPCLKCON);
+
return 0;
}
#endif
{
struct platform_device *pdev = to_platform_device(dev);
struct s3c_fb *sfb = platform_get_drvdata(pdev);
- struct s3c_fb_platdata *pd = sfb->pdata;
clk_enable(sfb->bus_clk);
clk_enable(sfb->lcd_clk);
/* setup gpio and output polarity controls */
- pd->setup_gpio();
- writel(pd->vidcon1, sfb->regs + VIDCON1);
+ s3c_fb_enable_fimd_bypass_disp1();
return 0;
}
NULL)
};
+#ifdef CONFIG_OF
+static const struct of_device_id exynos_fimd_match[] = {
+ { .compatible = "samsung,s3c-fb" },
+ {},
+};
+MODULE_DEVICE_TABLE(of, exynos_fimd_match);
+#endif
+
static struct platform_driver s3c_fb_driver = {
.probe = s3c_fb_probe,
.remove = __devexit_p(s3c_fb_remove),
.name = "s3c-fb",
.owner = THIS_MODULE,
.pm = &s3cfb_pm_ops,
+ .of_match_table = of_match_ptr(exynos_fimd_match),
},
};
--- /dev/null
+/* linux/drivers/video/s5p_mipi_dsi.c
+ *
+ * Samsung SoC MIPI-DSIM driver.
+ *
+ * Copyright (c) 2011 Samsung Electronics
+ *
+ * InKi Dae, <inki.dae@samsung.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <linux/clk.h>
+#include <linux/mutex.h>
+#include <linux/wait.h>
+#include <linux/fs.h>
+#include <linux/mm.h>
+#include <linux/fb.h>
+#include <linux/ctype.h>
+#include <linux/platform_device.h>
+#include <linux/io.h>
+#include <linux/irq.h>
+#include <linux/memory.h>
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/kthread.h>
+#include <linux/regulator/consumer.h>
+#include <linux/notifier.h>
+#include <linux/pm_runtime.h>
+
+#include <linux/gpio.h>
+
+#include <video/mipi_display.h>
+
+#include <plat/fb.h>
+#include <plat/regs-mipidsim.h>
+#include <plat/dsim.h>
+
+#include <mach/map.h>
+
+#include "s5p_mipi_dsi_lowlevel.h"
+#include "s5p_mipi_dsi.h"
+
+#ifdef CONFIG_HAS_EARLYSUSPEND
+#include <linux/earlysuspend.h>
+#endif
+
+static unsigned int dpll_table[15] = {
+ 100, 120, 170, 220, 270,
+ 320, 390, 450, 510, 560,
+ 640, 690, 770, 870, 950 };
+
+int s5p_mipi_dsi_wait_int_status(struct mipi_dsim_device *dsim,
+ unsigned int intSrc)
+{
+ while (1) {
+ if ((s5p_mipi_dsi_get_int_status(dsim) & intSrc)) {
+ s5p_mipi_dsi_clear_int_status(dsim, intSrc);
+ return 1;
+ } else if ((s5p_mipi_dsi_get_FIFOCTRL_status(dsim)
+ & 0xf00000) == 0)
+ return 0;
+ }
+}
+
+static int s5p_mipi_dsi_fb_notifier_callback(struct notifier_block *self,
+ unsigned long event, void *data)
+{
+ struct mipi_dsim_device *dsim;
+
+ if (event != FB_EVENT_RESUME)
+ return 0;
+
+ dsim = container_of(self, struct mipi_dsim_device, fb_notif);
+ s5p_mipi_dsi_func_reset(dsim);
+
+ return 0;
+}
+
+static int s5p_mipi_dsi_register_fb(struct mipi_dsim_device *dsim)
+{
+ memset(&dsim->fb_notif, 0, sizeof(dsim->fb_notif));
+ dsim->fb_notif.notifier_call = s5p_mipi_dsi_fb_notifier_callback;
+
+ return fb_register_client(&dsim->fb_notif);
+}
+
+static void s5p_mipi_dsi_long_data_wr(struct mipi_dsim_device *dsim,
+ unsigned int data0, unsigned int data1)
+{
+ unsigned int data_cnt = 0, payload = 0;
+
+ /* in case that data count is more then 4 */
+ for (data_cnt = 0; data_cnt < data1; data_cnt += 4) {
+ /*
+ * after sending 4bytes per one time,
+ * send remainder data less then 4.
+ */
+ if ((data1 - data_cnt) < 4) {
+ if ((data1 - data_cnt) == 3) {
+ payload = *(u8 *)(data0 + data_cnt) |
+ (*(u8 *)(data0 + (data_cnt + 1))) << 8 |
+ (*(u8 *)(data0 + (data_cnt + 2))) << 16;
+ dev_dbg(dsim->dev, "count = 3 payload = %x, %x %x %x\n",
+ payload, *(u8 *)(data0 + data_cnt),
+ *(u8 *)(data0 + (data_cnt + 1)),
+ *(u8 *)(data0 + (data_cnt + 2)));
+ } else if ((data1 - data_cnt) == 2) {
+ payload = *(u8 *)(data0 + data_cnt) |
+ (*(u8 *)(data0 + (data_cnt + 1))) << 8;
+ dev_dbg(dsim->dev,
+ "count = 2 payload = %x, %x %x\n", payload,
+ *(u8 *)(data0 + data_cnt),
+ *(u8 *)(data0 + (data_cnt + 1)));
+ } else if ((data1 - data_cnt) == 1) {
+ payload = *(u8 *)(data0 + data_cnt);
+ }
+
+ s5p_mipi_dsi_wr_tx_data(dsim, payload);
+ /* send 4bytes per one time. */
+ } else {
+ payload = *(u8 *)(data0 + data_cnt) |
+ (*(u8 *)(data0 + (data_cnt + 1))) << 8 |
+ (*(u8 *)(data0 + (data_cnt + 2))) << 16 |
+ (*(u8 *)(data0 + (data_cnt + 3))) << 24;
+
+ dev_dbg(dsim->dev,
+ "count = 4 payload = %x, %x %x %x %x\n",
+ payload, *(u8 *)(data0 + data_cnt),
+ *(u8 *)(data0 + (data_cnt + 1)),
+ *(u8 *)(data0 + (data_cnt + 2)),
+ *(u8 *)(data0 + (data_cnt + 3)));
+
+ s5p_mipi_dsi_wr_tx_data(dsim, payload);
+ }
+ }
+}
+
+int s5p_mipi_dsi_wr_data(struct mipi_dsim_device *dsim, unsigned int data_id,
+ unsigned int data0, unsigned int data1)
+{
+ unsigned long delay_val, udelay;
+ unsigned int check_rx_ack = 0;
+
+ if (dsim->state == DSIM_STATE_ULPS) {
+ dev_err(dsim->dev, "state is ULPS.\n");
+
+ return -EINVAL;
+ }
+
+ delay_val = MHZ / dsim->dsim_config->esc_clk;
+ udelay = 10 * delay_val;
+
+ mdelay(udelay);
+
+ switch (data_id) {
+ /* short packet types of packet types for command. */
+ case MIPI_DSI_GENERIC_SHORT_WRITE_0_PARAM:
+ case MIPI_DSI_GENERIC_SHORT_WRITE_1_PARAM:
+ case MIPI_DSI_GENERIC_SHORT_WRITE_2_PARAM:
+ case MIPI_DSI_DCS_SHORT_WRITE:
+ case MIPI_DSI_DCS_SHORT_WRITE_PARAM:
+ case MIPI_DSI_SET_MAXIMUM_RETURN_PACKET_SIZE:
+ s5p_mipi_dsi_wr_tx_header(dsim, data_id, data0, data1);
+ if (check_rx_ack)
+ /* process response func should be implemented */
+ return 0;
+ else
+ return -EINVAL;
+
+ /* general command */
+ case MIPI_DSI_COLOR_MODE_OFF:
+ case MIPI_DSI_COLOR_MODE_ON:
+ case MIPI_DSI_SHUTDOWN_PERIPHERAL:
+ case MIPI_DSI_TURN_ON_PERIPHERAL:
+ s5p_mipi_dsi_wr_tx_header(dsim, data_id, data0, data1);
+ if (check_rx_ack)
+ /* process response func should be implemented. */
+ return 0;
+ else
+ return -EINVAL;
+
+ /* packet types for video data */
+ case MIPI_DSI_V_SYNC_START:
+ case MIPI_DSI_V_SYNC_END:
+ case MIPI_DSI_H_SYNC_START:
+ case MIPI_DSI_H_SYNC_END:
+ case MIPI_DSI_END_OF_TRANSMISSION:
+ return 0;
+
+ /* short and response packet types for command */
+ case MIPI_DSI_GENERIC_READ_REQUEST_0_PARAM:
+ case MIPI_DSI_GENERIC_READ_REQUEST_1_PARAM:
+ case MIPI_DSI_GENERIC_READ_REQUEST_2_PARAM:
+ case MIPI_DSI_DCS_READ:
+ s5p_mipi_dsi_clear_all_interrupt(dsim);
+ s5p_mipi_dsi_wr_tx_header(dsim, data_id, data0, data1);
+ /* process response func should be implemented. */
+ return 0;
+
+ /* long packet type and null packet */
+ case MIPI_DSI_NULL_PACKET:
+ case MIPI_DSI_BLANKING_PACKET:
+ return 0;
+ case MIPI_DSI_GENERIC_LONG_WRITE:
+ case MIPI_DSI_DCS_LONG_WRITE:
+ {
+ unsigned int size, data_cnt = 0, payload = 0;
+
+ size = data1 * 4;
+
+ /* if data count is less then 4, then send 3bytes data. */
+ if (data1 < 4) {
+ payload = *(u8 *)(data0) |
+ *(u8 *)(data0 + 1) << 8 |
+ *(u8 *)(data0 + 2) << 16;
+
+ s5p_mipi_dsi_wr_tx_data(dsim, payload);
+
+ dev_dbg(dsim->dev, "count = %d payload = %x,%x %x %x\n",
+ data1, payload,
+ *(u8 *)(data0 + data_cnt),
+ *(u8 *)(data0 + (data_cnt + 1)),
+ *(u8 *)(data0 + (data_cnt + 2)));
+ /* in case that data count is more then 4 */
+ } else
+ s5p_mipi_dsi_long_data_wr(dsim, data0, data1);
+
+ /* put data into header fifo */
+ s5p_mipi_dsi_wr_tx_header(dsim, data_id, data1 & 0xff,
+ (data1 & 0xff00) >> 8);
+ }
+ if (s5p_mipi_dsi_wait_int_status(dsim, INTSRC_SFR_FIFO_EMPTY) == 0)
+ return -1;
+
+ if (check_rx_ack)
+ /* process response func should be implemented. */
+ return 0;
+ else
+ return -EINVAL;
+
+ /* packet typo for video data */
+ case MIPI_DSI_PACKED_PIXEL_STREAM_16:
+ case MIPI_DSI_PACKED_PIXEL_STREAM_18:
+ case MIPI_DSI_PIXEL_STREAM_3BYTE_18:
+ case MIPI_DSI_PACKED_PIXEL_STREAM_24:
+ if (check_rx_ack)
+ /* process response func should be implemented. */
+ return 0;
+ else
+ return -EINVAL;
+ default:
+ dev_warn(dsim->dev,
+ "data id %x is not supported current DSI spec.\n",
+ data_id);
+
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+int s5p_mipi_dsi_pll_on(struct mipi_dsim_device *dsim, unsigned int enable)
+{
+ int sw_timeout;
+
+ if (enable) {
+ sw_timeout = 1000;
+
+ s5p_mipi_dsi_clear_interrupt(dsim, INTSRC_PLL_STABLE);
+ s5p_mipi_dsi_enable_pll(dsim, 1);
+ while (1) {
+ sw_timeout--;
+ if (s5p_mipi_dsi_is_pll_stable(dsim))
+ return 0;
+ if (sw_timeout == 0)
+ return -EINVAL;
+ }
+ } else
+ s5p_mipi_dsi_enable_pll(dsim, 0);
+
+ return 0;
+}
+
+unsigned long s5p_mipi_dsi_change_pll(struct mipi_dsim_device *dsim,
+ unsigned int pre_divider, unsigned int main_divider,
+ unsigned int scaler)
+{
+ unsigned long dfin_pll, dfvco, dpll_out;
+ unsigned int i, freq_band = 0xf;
+
+ dfin_pll = (FIN_HZ / pre_divider);
+
+ if (dfin_pll < DFIN_PLL_MIN_HZ || dfin_pll > DFIN_PLL_MAX_HZ) {
+ dev_warn(dsim->dev, "fin_pll range should be 6MHz ~ 12MHz\n");
+ s5p_mipi_dsi_enable_afc(dsim, 0, 0);
+ } else {
+ if (dfin_pll < 7 * MHZ)
+ s5p_mipi_dsi_enable_afc(dsim, 1, 0x1);
+ else if (dfin_pll < 8 * MHZ)
+ s5p_mipi_dsi_enable_afc(dsim, 1, 0x0);
+ else if (dfin_pll < 9 * MHZ)
+ s5p_mipi_dsi_enable_afc(dsim, 1, 0x3);
+ else if (dfin_pll < 10 * MHZ)
+ s5p_mipi_dsi_enable_afc(dsim, 1, 0x2);
+ else if (dfin_pll < 11 * MHZ)
+ s5p_mipi_dsi_enable_afc(dsim, 1, 0x5);
+ else
+ s5p_mipi_dsi_enable_afc(dsim, 1, 0x4);
+ }
+
+ dfvco = dfin_pll * main_divider;
+ dev_dbg(dsim->dev, "dfvco = %lu, dfin_pll = %lu, main_divider = %d\n",
+ dfvco, dfin_pll, main_divider);
+ if (dfvco < DFVCO_MIN_HZ || dfvco > DFVCO_MAX_HZ)
+ dev_warn(dsim->dev, "fvco range should be 500MHz ~ 1000MHz\n");
+
+ dpll_out = dfvco / (1 << scaler);
+ dev_dbg(dsim->dev, "dpll_out = %lu, dfvco = %lu, scaler = %d\n",
+ dpll_out, dfvco, scaler);
+
+ for (i = 0; i < ARRAY_SIZE(dpll_table); i++) {
+ if (dpll_out < dpll_table[i] * MHZ) {
+ freq_band = i;
+ break;
+ }
+ }
+
+ dev_dbg(dsim->dev, "freq_band = %d\n", freq_band);
+
+ s5p_mipi_dsi_pll_freq(dsim, pre_divider, main_divider, scaler);
+
+ s5p_mipi_dsi_hs_zero_ctrl(dsim, 0);
+ s5p_mipi_dsi_prep_ctrl(dsim, 0);
+
+ /* Freq Band */
+ s5p_mipi_dsi_pll_freq_band(dsim, freq_band);
+
+ /* Stable time */
+ s5p_mipi_dsi_pll_stable_time(dsim, dsim->dsim_config->pll_stable_time);
+
+ /* Enable PLL */
+ dev_dbg(dsim->dev, "FOUT of mipi dphy pll is %luMHz\n",
+ (dpll_out / MHZ));
+
+ return dpll_out;
+}
+
+int s5p_mipi_dsi_set_clock(struct mipi_dsim_device *dsim,
+ unsigned int byte_clk_sel, unsigned int enable)
+{
+ unsigned int esc_div;
+ unsigned long esc_clk_error_rate;
+
+ if (enable) {
+ dsim->e_clk_src = byte_clk_sel;
+
+ /* Escape mode clock and byte clock source */
+ s5p_mipi_dsi_set_byte_clock_src(dsim, byte_clk_sel);
+
+ /* DPHY, DSIM Link : D-PHY clock out */
+ if (byte_clk_sel == DSIM_PLL_OUT_DIV8) {
+ dsim->hs_clk = s5p_mipi_dsi_change_pll(dsim,
+ dsim->dsim_config->p, dsim->dsim_config->m,
+ dsim->dsim_config->s);
+ if (dsim->hs_clk == 0) {
+ dev_err(dsim->dev,
+ "failed to get hs clock.\n");
+ return -EINVAL;
+ }
+
+ dsim->byte_clk = dsim->hs_clk / 8;
+ s5p_mipi_dsi_enable_pll_bypass(dsim, 0);
+ s5p_mipi_dsi_pll_on(dsim, 1);
+ /* DPHY : D-PHY clock out, DSIM link : external clock out */
+ } else if (byte_clk_sel == DSIM_EXT_CLK_DIV8)
+ dev_warn(dsim->dev,
+ "this project is not support \
+ external clock source for MIPI DSIM\n");
+ else if (byte_clk_sel == DSIM_EXT_CLK_BYPASS)
+ dev_warn(dsim->dev,
+ "this project is not support \
+ external clock source for MIPI DSIM\n");
+
+ /* escape clock divider */
+ esc_div = dsim->byte_clk / (dsim->dsim_config->esc_clk);
+ dev_dbg(dsim->dev,
+ "esc_div = %d, byte_clk = %lu, esc_clk = %lu\n",
+ esc_div, dsim->byte_clk, dsim->dsim_config->esc_clk);
+ if ((dsim->byte_clk / esc_div) >= (20 * MHZ) ||
+ (dsim->byte_clk / esc_div) >
+ dsim->dsim_config->esc_clk)
+ esc_div += 1;
+
+ dsim->escape_clk = dsim->byte_clk / esc_div;
+ dev_dbg(dsim->dev,
+ "escape_clk = %lu, byte_clk = %lu, esc_div = %d\n",
+ dsim->escape_clk, dsim->byte_clk, esc_div);
+
+ /* enable escape clock. */
+ s5p_mipi_dsi_enable_byte_clock(dsim, DSIM_ESCCLK_ON);
+
+ /* enable byte clk and escape clock */
+ s5p_mipi_dsi_set_esc_clk_prs(dsim, 1, esc_div);
+ /* escape clock on lane */
+ s5p_mipi_dsi_enable_esc_clk_on_lane(dsim,
+ (DSIM_LANE_CLOCK | dsim->data_lane), 1);
+
+ dev_dbg(dsim->dev, "byte clock is %luMHz\n",
+ (dsim->byte_clk / MHZ));
+ dev_dbg(dsim->dev, "escape clock that user's need is %lu\n",
+ (dsim->dsim_config->esc_clk / MHZ));
+ dev_dbg(dsim->dev, "escape clock divider is %x\n", esc_div);
+ dev_dbg(dsim->dev, "escape clock is %luMHz\n",
+ ((dsim->byte_clk / esc_div) / MHZ));
+
+ if ((dsim->byte_clk / esc_div) > dsim->escape_clk) {
+ esc_clk_error_rate = dsim->escape_clk /
+ (dsim->byte_clk / esc_div);
+ dev_warn(dsim->dev, "error rate is %lu over.\n",
+ (esc_clk_error_rate / 100));
+ } else if ((dsim->byte_clk / esc_div) < (dsim->escape_clk)) {
+ esc_clk_error_rate = (dsim->byte_clk / esc_div) /
+ dsim->escape_clk;
+ dev_warn(dsim->dev, "error rate is %lu under.\n",
+ (esc_clk_error_rate / 100));
+ }
+ } else {
+ s5p_mipi_dsi_enable_esc_clk_on_lane(dsim,
+ (DSIM_LANE_CLOCK | dsim->data_lane), 0);
+ s5p_mipi_dsi_set_esc_clk_prs(dsim, 0, 0);
+
+ /* disable escape clock. */
+ s5p_mipi_dsi_enable_byte_clock(dsim, DSIM_ESCCLK_OFF);
+
+ if (byte_clk_sel == DSIM_PLL_OUT_DIV8)
+ s5p_mipi_dsi_pll_on(dsim, 0);
+ }
+
+ return 0;
+}
+
+void s5p_mipi_dsi_d_phy_onoff(struct mipi_dsim_device *dsim,
+ unsigned int enable)
+{
+ if (dsim->pd->init_d_phy)
+ dsim->pd->init_d_phy(dsim, enable);
+}
+
+int s5p_mipi_dsi_init_dsim(struct mipi_dsim_device *dsim)
+{
+ s5p_mipi_dsi_d_phy_onoff(dsim, 1);
+
+ dsim->state = DSIM_STATE_INIT;
+
+ switch (dsim->dsim_config->e_no_data_lane) {
+ case DSIM_DATA_LANE_1:
+ dsim->data_lane = DSIM_LANE_DATA0;
+ break;
+ case DSIM_DATA_LANE_2:
+ dsim->data_lane = DSIM_LANE_DATA0 | DSIM_LANE_DATA1;
+ break;
+ case DSIM_DATA_LANE_3:
+ dsim->data_lane = DSIM_LANE_DATA0 | DSIM_LANE_DATA1 |
+ DSIM_LANE_DATA2;
+ break;
+ case DSIM_DATA_LANE_4:
+ dsim->data_lane = DSIM_LANE_DATA0 | DSIM_LANE_DATA1 |
+ DSIM_LANE_DATA2 | DSIM_LANE_DATA3;
+ break;
+ default:
+ dev_info(dsim->dev, "data lane is invalid.\n");
+ return -EINVAL;
+ };
+
+ s5p_mipi_dsi_sw_reset(dsim);
+ s5p_mipi_dsi_dp_dn_swap(dsim, 0);
+
+ return 0;
+}
+
+int s5p_mipi_dsi_enable_frame_done_int(struct mipi_dsim_device *dsim,
+ unsigned int enable)
+{
+ /* enable only frame done interrupt */
+ s5p_mipi_dsi_set_interrupt_mask(dsim, INTMSK_FRAME_DONE, enable);
+
+ return 0;
+}
+
+int s5p_mipi_dsi_set_display_mode(struct mipi_dsim_device *dsim,
+ struct mipi_dsim_config *dsim_config)
+{
+ struct fb_videomode *lcd_video = NULL;
+ struct s3c_fb_pd_win *pd;
+ unsigned int width = 0, height = 0;
+ pd = (struct s3c_fb_pd_win *)dsim->dsim_config->lcd_panel_info;
+ lcd_video = (struct fb_videomode *)&pd->win_mode;
+
+ width = dsim->pd->dsim_lcd_config->lcd_size.width;
+ height = dsim->pd->dsim_lcd_config->lcd_size.height;
+
+ /* in case of VIDEO MODE (RGB INTERFACE) */
+ if (dsim->dsim_config->e_interface == (u32) DSIM_VIDEO) {
+ s5p_mipi_dsi_set_main_disp_vporch(dsim,
+ DSIM_CMD_LEN,
+ dsim->pd->dsim_lcd_config->rgb_timing.left_margin,
+ dsim->pd->dsim_lcd_config->rgb_timing.right_margin);
+ s5p_mipi_dsi_set_main_disp_hporch(dsim,
+ dsim->pd->dsim_lcd_config->rgb_timing.upper_margin,
+ dsim->pd->dsim_lcd_config->rgb_timing.lower_margin);
+ s5p_mipi_dsi_set_main_disp_sync_area(dsim,
+ dsim->pd->dsim_lcd_config->rgb_timing.vsync_len,
+ dsim->pd->dsim_lcd_config->rgb_timing.hsync_len);
+ }
+ s5p_mipi_dsi_set_main_disp_resol(dsim, height, width);
+ s5p_mipi_dsi_display_config(dsim);
+ return 0;
+}
+
+int s5p_mipi_dsi_init_link(struct mipi_dsim_device *dsim)
+{
+ unsigned int time_out = 100;
+ unsigned int id;
+ id = dsim->id;
+ switch (dsim->state) {
+ case DSIM_STATE_INIT:
+ s5p_mipi_dsi_sw_reset(dsim);
+ s5p_mipi_dsi_init_fifo_pointer(dsim, 0x1f);
+
+ /* dsi configuration */
+ s5p_mipi_dsi_init_config(dsim);
+ s5p_mipi_dsi_enable_lane(dsim, DSIM_LANE_CLOCK, 1);
+ s5p_mipi_dsi_enable_lane(dsim, dsim->data_lane, 1);
+
+ /* set clock configuration */
+ s5p_mipi_dsi_set_clock(dsim, dsim->dsim_config->e_byte_clk, 1);
+
+ mdelay(100);
+
+ /* check clock and data lane state are stop state */
+ while (!(s5p_mipi_dsi_is_lane_state(dsim))) {
+ time_out--;
+ if (time_out == 0) {
+ dev_err(dsim->dev,
+ "DSI Master is not stop state.\n");
+ dev_err(dsim->dev,
+ "Check initialization process\n");
+
+ return -EINVAL;
+ }
+ }
+
+ if (time_out != 0) {
+ dev_info(dsim->dev,
+ "DSI Master driver has been completed.\n");
+ dev_info(dsim->dev, "DSI Master state is stop state\n");
+ }
+
+ dsim->state = DSIM_STATE_STOP;
+
+ /* BTA sequence counters */
+ s5p_mipi_dsi_set_stop_state_counter(dsim,
+ dsim->dsim_config->stop_holding_cnt);
+ s5p_mipi_dsi_set_bta_timeout(dsim,
+ dsim->dsim_config->bta_timeout);
+ s5p_mipi_dsi_set_lpdr_timeout(dsim,
+ dsim->dsim_config->rx_timeout);
+
+ return 0;
+ default:
+ dev_info(dsim->dev, "DSI Master is already init.\n");
+ return 0;
+ }
+
+ return 0;
+}
+
+int s5p_mipi_dsi_set_hs_enable(struct mipi_dsim_device *dsim)
+{
+ if (dsim->state == DSIM_STATE_STOP) {
+ if (dsim->e_clk_src != DSIM_EXT_CLK_BYPASS) {
+ dsim->state = DSIM_STATE_HSCLKEN;
+
+ /* set LCDC and CPU transfer mode to HS. */
+ s5p_mipi_dsi_set_lcdc_transfer_mode(dsim, 0);
+ s5p_mipi_dsi_set_cpu_transfer_mode(dsim, 0);
+
+ s5p_mipi_dsi_enable_hs_clock(dsim, 1);
+
+ return 0;
+ } else
+ dev_warn(dsim->dev,
+ "clock source is external bypass.\n");
+ } else
+ dev_warn(dsim->dev, "DSIM is not stop state.\n");
+
+ return 0;
+}
+
+int s5p_mipi_dsi_set_data_transfer_mode(struct mipi_dsim_device *dsim,
+ unsigned int mode)
+{
+ if (mode) {
+ if (dsim->state != DSIM_STATE_HSCLKEN) {
+ dev_err(dsim->dev, "HS Clock lane is not enabled.\n");
+ return -EINVAL;
+ }
+
+ s5p_mipi_dsi_set_lcdc_transfer_mode(dsim, 0);
+ } else {
+ if (dsim->state == DSIM_STATE_INIT || dsim->state ==
+ DSIM_STATE_ULPS) {
+ dev_err(dsim->dev,
+ "DSI Master is not STOP or HSDT state.\n");
+ return -EINVAL;
+ }
+
+ s5p_mipi_dsi_set_cpu_transfer_mode(dsim, 0);
+ }
+ return 0;
+}
+
+int s5p_mipi_dsi_get_frame_done_status(struct mipi_dsim_device *dsim)
+{
+ return _s5p_mipi_dsi_get_frame_done_status(dsim);
+}
+
+int s5p_mipi_dsi_clear_frame_done(struct mipi_dsim_device *dsim)
+{
+ _s5p_mipi_dsi_clear_frame_done(dsim);
+
+ return 0;
+}
+
+static irqreturn_t s5p_mipi_dsi_interrupt_handler(int irq, void *dev_id)
+{
+ unsigned int int_src;
+ struct mipi_dsim_device *dsim = dev_id;
+
+ s5p_mipi_dsi_set_interrupt_mask(dsim, 0xffffffff, 1);
+
+ int_src = readl(dsim->reg_base + S5P_DSIM_INTSRC);
+ s5p_mipi_dsi_clear_interrupt(dsim, int_src);
+
+ if (!(int_src & (INTSRC_PLL_STABLE | INTSRC_FRAME_DONE)))
+ printk(KERN_ERR "mipi dsi interrupt source (%x).\n", int_src);
+
+ s5p_mipi_dsi_set_interrupt_mask(dsim, 0xffffffff, 0);
+ return IRQ_HANDLED;
+}
+
+static inline void s5p_mipi_initialize_mipi_client(struct device *dev,
+ struct mipi_dsim_device *dsim)
+{
+ int again = 1;
+ while (again == 1) {
+ pm_runtime_get_sync(dev);
+ clk_enable(dsim->clock);
+ s5p_mipi_dsi_init_dsim(dsim);
+ s5p_mipi_dsi_init_link(dsim);
+ s5p_mipi_dsi_enable_hs_clock(dsim, 1);
+ s5p_mipi_dsi_set_cpu_transfer_mode(dsim, 1);
+ s5p_mipi_dsi_set_display_mode(dsim, dsim->dsim_config);
+ s5p_mipi_dsi_clear_int_status(dsim, INTSRC_SFR_FIFO_EMPTY);
+ if (dsim->dsim_lcd_drv->displayon(dsim) == 0)
+ again = 1;
+ else
+ again = 0;
+ s5p_mipi_dsi_set_cpu_transfer_mode(dsim, 0);
+ if (again == 1)
+ s5p_mipi_dsi_sw_reset(dsim);
+ }
+}
+
+#ifdef CONFIG_PM
+#ifdef CONFIG_HAS_EARLYSUSPEND
+static void s5p_mipi_dsi_early_suspend(struct early_suspend *handler)
+{
+ struct mipi_dsim_device *dsim =
+ container_of(handler, struct mipi_dsim_device, early_suspend);
+ struct platform_device *pdev = to_platform_device(dsim->dev);
+
+ dsim->dsim_lcd_drv->suspend(dsim);
+ s5p_mipi_dsi_d_phy_onoff(dsim, 0);
+ clk_disable(dsim->clock);
+ pm_runtime_put_sync(&pdev->dev);
+}
+
+static void s5p_mipi_dsi_late_resume(struct early_suspend *handler)
+{
+ struct mipi_dsim_device *dsim =
+ container_of(handler, struct mipi_dsim_device, early_suspend);
+ struct platform_device *pdev = to_platform_device(dsim->dev);
+ s5p_mipi_initialize_mipi_client(&pdev->dev, dsim);
+}
+#else
+static int s5p_mipi_dsi_suspend(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct mipi_dsim_device *dsim = platform_get_drvdata(pdev);
+
+ dsim->dsim_lcd_drv->suspend(dsim);
+ s5p_mipi_dsi_d_phy_onoff(dsim, 0);
+ clk_disable(dsim->clock);
+ pm_runtime_put_sync(dev);
+ return 0;
+}
+
+static int s5p_mipi_dsi_resume(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct mipi_dsim_device *dsim = platform_get_drvdata(pdev);
+
+ return 0;
+ s5p_mipi_initialize_mipi_client(dev, dsim);
+}
+#endif
+static int s5p_mipi_dsi_runtime_suspend(struct device *dev)
+{
+ return 0;
+}
+
+static int s5p_mipi_dsi_runtime_resume(struct device *dev)
+{
+ return 0;
+}
+#else
+#define s5p_mipi_dsi_suspend NULL
+#define s5p_mipi_dsi_resume NULL
+#define s5p_mipi_dsi_runtime_suspend NULL
+#define s5p_mipi_dsi_runtime_resume NULL
+#endif
+
+
+static int s5p_mipi_dsi_probe(struct platform_device *pdev)
+{
+ struct resource *res;
+ struct mipi_dsim_device *dsim = NULL;
+ struct mipi_dsim_config *dsim_config;
+ struct s5p_platform_mipi_dsim *dsim_pd;
+ int ret = -1;
+
+ if (!dsim)
+ dsim = kzalloc(sizeof(struct mipi_dsim_device),
+ GFP_KERNEL);
+ if (!dsim) {
+ dev_err(&pdev->dev, "failed to allocate dsim object.\n");
+ return -EFAULT;
+ }
+
+ dsim->pd = to_dsim_plat(&pdev->dev);
+ dsim->dev = &pdev->dev;
+ dsim->id = pdev->id;
+
+ ret = s5p_mipi_dsi_register_fb(dsim);
+ if (ret) {
+ dev_err(&pdev->dev, "failed to register fb notifier chain\n");
+ return -EFAULT;
+ }
+
+ pm_runtime_enable(&pdev->dev);
+
+ /* get s5p_platform_mipi_dsim. */
+ dsim_pd = (struct s5p_platform_mipi_dsim *)dsim->pd;
+ /* get mipi_dsim_config. */
+ dsim_config = dsim_pd->dsim_config;
+ dsim->dsim_config = dsim_config;
+
+ dsim->clock = clk_get(&pdev->dev, dsim->pd->clk_name);
+ if (IS_ERR(dsim->clock)) {
+ dev_err(&pdev->dev, "failed to get dsim clock source\n");
+ goto err_clock_get;
+ }
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!res) {
+ dev_err(&pdev->dev, "failed to get io memory region\n");
+ ret = -EINVAL;
+ goto err_platform_get;
+ }
+ res = request_mem_region(res->start, resource_size(res),
+ dev_name(&pdev->dev));
+ if (!res) {
+ dev_err(&pdev->dev, "failed to request io memory region\n");
+ ret = -EINVAL;
+ goto err_mem_region;
+ }
+
+ dsim->res = res;
+ dsim->reg_base = ioremap(res->start, resource_size(res));
+ if (!dsim->reg_base) {
+ dev_err(&pdev->dev, "failed to remap io region\n");
+ ret = -EINVAL;
+ goto err_mem_region;
+ }
+
+ /*
+ * it uses frame done interrupt handler
+ * only in case of MIPI Video mode.
+ */
+ if (dsim->pd->dsim_config->e_interface == DSIM_VIDEO) {
+ dsim->irq = platform_get_irq(pdev, 0);
+ if (request_irq(dsim->irq, s5p_mipi_dsi_interrupt_handler,
+ IRQF_DISABLED, "mipi-dsi", dsim)) {
+ dev_err(&pdev->dev, "request_irq failed.\n");
+ goto err_irq;
+ }
+ }
+
+ dsim->dsim_lcd_drv = dsim->dsim_config->dsim_ddi_pd;
+
+ if (dsim->dsim_config == NULL) {
+ dev_err(&pdev->dev, "dsim_config is NULL.\n");
+ goto err_dsim_config;
+ }
+
+ dsim->dsim_lcd_drv->probe(dsim);
+
+ s5p_mipi_initialize_mipi_client(&pdev->dev, dsim);
+
+ dev_info(&pdev->dev, "mipi-dsi driver(%s mode) has been probed.\n",
+ (dsim_config->e_interface == DSIM_COMMAND) ?
+ "CPU" : "RGB");
+
+#ifdef CONFIG_HAS_EARLYSUSPEND
+ dsim->early_suspend.suspend = s5p_mipi_dsi_early_suspend;
+ dsim->early_suspend.resume = s5p_mipi_dsi_late_resume;
+ dsim->early_suspend.level = EARLY_SUSPEND_LEVEL_DISABLE_FB + 1;
+ register_early_suspend(&(dsim->early_suspend));
+#endif
+ platform_set_drvdata(pdev, dsim);
+
+ return 0;
+
+err_dsim_config:
+err_irq:
+ release_resource(dsim->res);
+ kfree(dsim->res);
+
+ iounmap((void __iomem *) dsim->reg_base);
+
+err_mem_region:
+err_platform_get:
+ clk_disable(dsim->clock);
+ clk_put(dsim->clock);
+
+err_clock_get:
+ kfree(dsim);
+ pm_runtime_put_sync(&pdev->dev);
+ return ret;
+
+}
+
+static int __devexit s5p_mipi_dsi_remove(struct platform_device *pdev)
+{
+ struct mipi_dsim_device *dsim = platform_get_drvdata(pdev);
+
+ if (dsim->dsim_config->e_interface == DSIM_VIDEO)
+ free_irq(dsim->irq, dsim);
+
+ iounmap(dsim->reg_base);
+
+ clk_disable(dsim->clock);
+ clk_put(dsim->clock);
+
+ release_resource(dsim->res);
+ kfree(dsim->res);
+
+ kfree(dsim);
+
+ return 0;
+}
+
+static const struct dev_pm_ops mipi_dsi_pm_ops = {
+#ifndef CONFIG_HAS_EARLYSUSPEND
+ .suspend = s5p_mipi_dsi_suspend,
+ .resume = s5p_mipi_dsi_resume,
+#endif
+ .runtime_suspend = s5p_mipi_dsi_runtime_suspend,
+ .runtime_resume = s5p_mipi_dsi_runtime_resume,
+};
+
+#ifdef CONFIG_OF
+ static const struct of_device_id exynos_mipi_match[] = {
+ { .compatible = "samsung,exynos-mipi" },
+ {},
+ };
+ MODULE_DEVICE_TABLE(of, exynos_mipi_match);
+#endif
+
+static struct platform_driver s5p_mipi_dsi_driver = {
+ .probe = s5p_mipi_dsi_probe,
+ .remove = __devexit_p(s5p_mipi_dsi_remove),
+ .driver = {
+ .name = "s5p-mipi-dsim",
+ .owner = THIS_MODULE,
+ .pm = &mipi_dsi_pm_ops,
+ .of_match_table = of_match_ptr(exynos_mipi_match),
+ },
+};
+
+static int s5p_mipi_dsi_register(void)
+{
+ platform_driver_register(&s5p_mipi_dsi_driver);
+
+ return 0;
+}
+
+static void s5p_mipi_dsi_unregister(void)
+{
+ platform_driver_unregister(&s5p_mipi_dsi_driver);
+}
+module_init(s5p_mipi_dsi_register);
+module_exit(s5p_mipi_dsi_unregister);
+
+MODULE_AUTHOR("InKi Dae <inki.dae@samsung.com>");
+MODULE_DESCRIPTION("Samusung MIPI-DSI driver");
+MODULE_LICENSE("GPL");
--- /dev/null
+/* linux/drivers/video/s5p_mipi_dsi.h
+ *
+ * Header file for Samsung MIPI-DSI common driver.
+ *
+ * Copyright (c) 2011 Samsung Electronics
+ * InKi Dae <inki.dae@samsung.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#ifndef _S5P_MIPI_DSI_H
+#define _S5P_MIPI_DSI_H
+
+/* MIPI-DSIM status types. */
+enum {
+ DSIM_STATE_INIT, /* should be initialized. */
+ DSIM_STATE_STOP, /* CPU and LCDC are LP mode. */
+ DSIM_STATE_HSCLKEN, /* HS clock was enabled. */
+ DSIM_STATE_ULPS
+};
+
+/* define DSI lane types. */
+enum {
+ DSIM_LANE_CLOCK = (1 << 0),
+ DSIM_LANE_DATA0 = (1 << 1),
+ DSIM_LANE_DATA1 = (1 << 2),
+ DSIM_LANE_DATA2 = (1 << 3),
+ DSIM_LANE_DATA3 = (1 << 4),
+};
+
+#define MHZ (1000 * 1000)
+#define FIN_HZ (24 * MHZ)
+
+#define DFIN_PLL_MIN_HZ (6 * MHZ)
+#define DFIN_PLL_MAX_HZ (12 * MHZ)
+
+#define DFVCO_MIN_HZ (500 * MHZ)
+#define DFVCO_MAX_HZ (1000 * MHZ)
+
+#define TRY_GET_FIFO_TIMEOUT (5000 * 2)
+
+#define DSIM_ESCCLK_ON (0x1)
+#define DSIM_ESCCLK_OFF (0x0)
+
+#define DSIM_CMD_LEN (0xf)
+
+int s5p_mipi_dsi_wr_data(struct mipi_dsim_device *dsim, unsigned int
+ data_id,
+ unsigned int data0, unsigned int data1);
+#endif /* _S5P_MIPI_DSI_H */
--- /dev/null
+/* linux/drivers/video/s5p_mipi_dsi_lowlevel.c
+ *
+ * Samsung MIPI-DSI lowlevel driver.
+ *
+ * Copyright (c) 2011 Samsung Electronics
+ *
+ * InKi Dae, <inki.dae@samsung.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <linux/mutex.h>
+#include <linux/wait.h>
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/mm.h>
+#include <linux/ctype.h>
+#include <linux/io.h>
+
+#include <mach/map.h>
+
+#include <plat/dsim.h>
+#include <plat/regs-mipidsim.h>
+
+void s5p_mipi_dsi_func_reset(struct mipi_dsim_device *dsim)
+{
+ unsigned int reg;
+
+ reg = readl(dsim->reg_base + S5P_DSIM_SWRST);
+
+ reg |= DSIM_FUNCRST;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_SWRST);
+}
+
+void s5p_mipi_dsi_sw_reset(struct mipi_dsim_device *dsim)
+{
+ unsigned int reg;
+
+ reg = readl(dsim->reg_base + S5P_DSIM_SWRST);
+
+ reg |= DSIM_SWRST;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_SWRST);
+}
+
+void s5p_mipi_dsi_set_interrupt_mask(struct mipi_dsim_device *dsim,
+ unsigned int mode, unsigned int mask)
+{
+ unsigned int reg = readl(dsim->reg_base + S5P_DSIM_INTMSK);
+
+ if (mask)
+ reg |= mode;
+ else
+ reg &= ~mode;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_INTMSK);
+}
+
+void s5p_mipi_dsi_init_fifo_pointer(struct mipi_dsim_device *dsim,
+ unsigned int cfg)
+{
+ unsigned int reg;
+
+ reg = readl(dsim->reg_base + S5P_DSIM_FIFOCTRL);
+
+ writel(reg & ~(cfg), dsim->reg_base + S5P_DSIM_FIFOCTRL);
+ mdelay(10);
+ reg |= cfg;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_FIFOCTRL);
+}
+
+/*
+ * this function set PLL P, M and S value in D-PHY
+ */
+void s5p_mipi_dsi_set_phy_tunning(struct mipi_dsim_device *dsim,
+ unsigned int value)
+{
+ writel(DSIM_AFC_CTL(value), dsim->reg_base + S5P_DSIM_PHYACCHR);
+}
+
+void s5p_mipi_dsi_set_main_disp_resol(struct mipi_dsim_device *dsim,
+ unsigned int vert_resol, unsigned int hori_resol)
+{
+ unsigned int reg;
+
+ /* standby should be set after configuration so set to not ready*/
+ reg = (readl(dsim->reg_base + S5P_DSIM_MDRESOL)) &
+ ~(DSIM_MAIN_STAND_BY);
+ writel(reg, dsim->reg_base + S5P_DSIM_MDRESOL);
+
+ reg &= ~(0x7ff << 16) & ~(0x7ff << 0);
+ reg |= DSIM_MAIN_VRESOL(vert_resol) | DSIM_MAIN_HRESOL(hori_resol);
+
+ reg |= DSIM_MAIN_STAND_BY;
+ writel(reg, dsim->reg_base + S5P_DSIM_MDRESOL);
+}
+
+void s5p_mipi_dsi_set_main_disp_vporch(struct mipi_dsim_device *dsim,
+ unsigned int cmd_allow, unsigned int vfront, unsigned int vback)
+{
+ unsigned int reg;
+
+ reg = (readl(dsim->reg_base + S5P_DSIM_MVPORCH)) &
+ ~(DSIM_CMD_ALLOW_MASK) & ~(DSIM_STABLE_VFP_MASK) &
+ ~(DSIM_MAIN_VBP_MASK);
+
+ reg |= ((cmd_allow & 0xf) << DSIM_CMD_ALLOW_SHIFT) |
+ ((vfront & 0x7ff) << DSIM_STABLE_VFP_SHIFT) |
+ ((vback & 0x7ff) << DSIM_MAIN_VBP_SHIFT);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_MVPORCH);
+}
+
+void s5p_mipi_dsi_set_main_disp_hporch(struct mipi_dsim_device *dsim,
+ unsigned int front, unsigned int back)
+{
+ unsigned int reg;
+
+ reg = (readl(dsim->reg_base + S5P_DSIM_MHPORCH)) &
+ ~(DSIM_MAIN_HFP_MASK) & ~(DSIM_MAIN_HBP_MASK);
+
+ reg |= (front << DSIM_MAIN_HFP_SHIFT) | (back << DSIM_MAIN_HBP_SHIFT);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_MHPORCH);
+}
+
+void s5p_mipi_dsi_set_main_disp_sync_area(struct mipi_dsim_device *dsim,
+ unsigned int vert, unsigned int hori)
+{
+ unsigned int reg;
+
+ reg = (readl(dsim->reg_base + S5P_DSIM_MSYNC)) &
+ ~(DSIM_MAIN_VSA_MASK) & ~(DSIM_MAIN_HSA_MASK);
+
+ reg |= ((vert & 0x3ff) << DSIM_MAIN_VSA_SHIFT) |
+ (hori << DSIM_MAIN_HSA_SHIFT);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_MSYNC);
+}
+
+void s5p_mipi_dsi_set_sub_disp_resol(struct mipi_dsim_device *dsim,
+ unsigned int vert, unsigned int hori)
+{
+ unsigned int reg;
+
+ reg = (readl(dsim->reg_base + S5P_DSIM_SDRESOL)) &
+ ~(DSIM_SUB_STANDY_MASK);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_SDRESOL);
+
+ reg &= ~(DSIM_SUB_VRESOL_MASK) | ~(DSIM_SUB_HRESOL_MASK);
+ reg |= ((vert & 0x7ff) << DSIM_SUB_VRESOL_SHIFT) |
+ ((hori & 0x7ff) << DSIM_SUB_HRESOL_SHIFT);
+ writel(reg, dsim->reg_base + S5P_DSIM_SDRESOL);
+
+ reg |= (1 << DSIM_SUB_STANDY_SHIFT);
+ writel(reg, dsim->reg_base + S5P_DSIM_SDRESOL);
+}
+
+void s5p_mipi_dsi_init_config(struct mipi_dsim_device *dsim)
+{
+ struct mipi_dsim_config *dsim_config = dsim->dsim_config;
+
+ unsigned int cfg = (readl(dsim->reg_base + S5P_DSIM_CONFIG)) &
+ ~(1 << 28) & ~(0x1f << 20) & ~(0x3 << 5);
+
+ cfg = (dsim_config->auto_flush << 29) |
+ (dsim_config->eot_disable << 28) |
+ (dsim_config->auto_vertical_cnt << DSIM_AUTO_MODE_SHIFT) |
+ (dsim_config->hse << DSIM_HSE_MODE_SHIFT) |
+ (dsim_config->hfp << DSIM_HFP_MODE_SHIFT) |
+ (dsim_config->hbp << DSIM_HBP_MODE_SHIFT) |
+ (dsim_config->hsa << DSIM_HSA_MODE_SHIFT) |
+ (dsim_config->e_no_data_lane << DSIM_NUM_OF_DATALANE_SHIFT);
+
+ writel(cfg, dsim->reg_base + S5P_DSIM_CONFIG);
+}
+
+void s5p_mipi_dsi_display_config(struct mipi_dsim_device *dsim)
+{
+ u32 reg = (readl(dsim->reg_base + S5P_DSIM_CONFIG)) &
+ ~(0x3 << 26) & ~(1 << 25) & ~(0x3 << 18) & ~(0x7 << 12) &
+ ~(0x3 << 16) & ~(0x7 << 8);
+
+ if (dsim->pd->dsim_config->e_interface == DSIM_VIDEO)
+ reg |= (1 << 25);
+ else if (dsim->pd->dsim_config->e_interface == DSIM_COMMAND)
+ reg &= ~(1 << 25);
+ else {
+ dev_err(dsim->dev, "this ddi is not MIPI interface.\n");
+ return;
+ }
+
+ /* main lcd */
+ reg |= ((u8) (dsim->pd->dsim_config->e_burst_mode) & 0x3) << 26 |
+ ((u8) (dsim->pd->dsim_config->e_virtual_ch) & 0x3) << 18 |
+ ((u8) (dsim->pd->dsim_config->e_pixel_format) & 0x7) << 12;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_CONFIG);
+}
+
+void s5p_mipi_dsi_enable_lane(struct mipi_dsim_device *dsim, unsigned int lane,
+ unsigned int enable)
+{
+ unsigned int reg;
+
+ reg = readl(dsim->reg_base + S5P_DSIM_CONFIG);
+
+ if (enable)
+ reg |= DSIM_LANE_ENx(lane);
+ else
+ reg &= ~DSIM_LANE_ENx(lane);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_CONFIG);
+}
+
+
+void s5p_mipi_dsi_set_data_lane_number(struct mipi_dsim_device *dsim,
+ unsigned int count)
+{
+ unsigned int cfg;
+
+ /* get the data lane number. */
+ cfg = DSIM_NUM_OF_DATA_LANE(count);
+
+ writel(cfg, dsim->reg_base + S5P_DSIM_CONFIG);
+}
+
+void s5p_mipi_dsi_enable_afc(struct mipi_dsim_device *dsim, unsigned int enable,
+ unsigned int afc_code)
+{
+ unsigned int reg = readl(dsim->reg_base + S5P_DSIM_PHYACCHR);
+
+ if (enable) {
+ reg |= (1 << 14);
+ reg &= ~(0x7 << 5);
+ reg |= (afc_code & 0x7) << 5;
+ } else
+ reg &= ~(1 << 14);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_PHYACCHR);
+}
+
+void s5p_mipi_dsi_enable_pll_bypass(struct mipi_dsim_device *dsim,
+ unsigned int enable)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_CLKCTRL)) &
+ ~(DSIM_PLL_BYPASS_EXTERNAL);
+
+ reg |= enable << DSIM_PLL_BYPASS_SHIFT;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_CLKCTRL);
+}
+
+void s5p_mipi_dsi_set_pll_pms(struct mipi_dsim_device *dsim, unsigned int p,
+ unsigned int m, unsigned int s)
+{
+ unsigned int reg = readl(dsim->reg_base + S5P_DSIM_PLLCTRL);
+
+ reg |= ((p & 0x3f) << 13) | ((m & 0x1ff) << 4) | ((s & 0x7) << 1);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_PLLCTRL);
+}
+
+void s5p_mipi_dsi_pll_freq_band(struct mipi_dsim_device *dsim,
+ unsigned int freq_band)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_PLLCTRL)) &
+ ~(0x1f << DSIM_FREQ_BAND_SHIFT);
+
+ reg |= ((freq_band & 0x1f) << DSIM_FREQ_BAND_SHIFT);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_PLLCTRL);
+}
+
+void s5p_mipi_dsi_pll_freq(struct mipi_dsim_device *dsim,
+ unsigned int pre_divider, unsigned int main_divider,
+ unsigned int scaler)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_PLLCTRL)) &
+ ~(0x7ffff << 1);
+
+ reg |= (pre_divider & 0x3f) << 13 | (main_divider & 0x1ff) << 4 |
+ (scaler & 0x7) << 1;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_PLLCTRL);
+}
+
+void s5p_mipi_dsi_pll_stable_time(struct mipi_dsim_device *dsim,
+ unsigned int lock_time)
+{
+ writel(lock_time, dsim->reg_base + S5P_DSIM_PLLTMR);
+}
+
+void s5p_mipi_dsi_enable_pll(struct mipi_dsim_device *dsim, unsigned int enable)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_PLLCTRL)) &
+ ~(0x1 << DSIM_PLL_EN_SHIFT);
+
+ reg |= ((enable & 0x1) << DSIM_PLL_EN_SHIFT);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_PLLCTRL);
+}
+
+void s5p_mipi_dsi_set_byte_clock_src(struct mipi_dsim_device *dsim,
+ unsigned int src)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_CLKCTRL)) &
+ ~(0x3 << DSIM_BYTE_CLK_SRC_SHIFT);
+
+ reg |= ((unsigned int) src) << DSIM_BYTE_CLK_SRC_SHIFT;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_CLKCTRL);
+}
+
+void s5p_mipi_dsi_enable_byte_clock(struct mipi_dsim_device *dsim,
+ unsigned int enable)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_CLKCTRL)) &
+ ~(1 << DSIM_BYTE_CLKEN_SHIFT);
+
+ reg |= enable << DSIM_BYTE_CLKEN_SHIFT;
+ writel(reg, dsim->reg_base + S5P_DSIM_CLKCTRL);
+}
+
+void s5p_mipi_dsi_set_esc_clk_prs(struct mipi_dsim_device *dsim,
+ unsigned int enable, unsigned int prs_val)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_CLKCTRL)) &
+ ~(1 << DSIM_ESC_CLKEN_SHIFT) & ~(0xffff);
+
+ reg |= enable << DSIM_ESC_CLKEN_SHIFT;
+ if (enable)
+ reg |= prs_val;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_CLKCTRL);
+}
+
+void s5p_mipi_dsi_enable_esc_clk_on_lane(struct mipi_dsim_device *dsim,
+ unsigned int lane_sel, unsigned int enable)
+{
+ unsigned int reg = readl(dsim->reg_base + S5P_DSIM_CLKCTRL);
+
+ if (enable)
+ reg |= DSIM_LANE_ESC_CLKEN(lane_sel);
+ else
+
+ reg &= ~DSIM_LANE_ESC_CLKEN(lane_sel);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_CLKCTRL);
+}
+
+void s5p_mipi_dsi_force_dphy_stop_state(struct mipi_dsim_device *dsim,
+ unsigned int enable)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_ESCMODE)) &
+ ~(0x1 << DSIM_FORCE_STOP_STATE_SHIFT);
+
+ reg |= ((enable & 0x1) << DSIM_FORCE_STOP_STATE_SHIFT);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_ESCMODE);
+}
+
+unsigned int s5p_mipi_dsi_is_lane_state(struct mipi_dsim_device *dsim)
+{
+ unsigned int reg = readl(dsim->reg_base + S5P_DSIM_STATUS);
+ /**
+ * check clock and data lane states.
+ * if MIPI-DSI controller was enabled at bootloader then
+ * TX_READY_HS_CLK is enabled otherwise STOP_STATE_CLK.
+ * so it should be checked for two case.
+ */
+ if ((reg & DSIM_STOP_STATE_DAT(0xf)) &&
+ ((reg & DSIM_STOP_STATE_CLK) ||
+ (reg & DSIM_TX_READY_HS_CLK)))
+ return 1;
+ else
+ return 0;
+
+ return 0;
+}
+
+void s5p_mipi_dsi_set_stop_state_counter(struct mipi_dsim_device *dsim,
+ unsigned int cnt_val)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_ESCMODE)) &
+ ~(0x7ff << DSIM_STOP_STATE_CNT_SHIFT);
+
+ reg |= ((cnt_val & 0x7ff) << DSIM_STOP_STATE_CNT_SHIFT);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_ESCMODE);
+}
+
+void s5p_mipi_dsi_set_bta_timeout(struct mipi_dsim_device *dsim,
+ unsigned int timeout)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_TIMEOUT)) &
+ ~(0xff << DSIM_BTA_TOUT_SHIFT);
+
+ reg |= (timeout << DSIM_BTA_TOUT_SHIFT);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_TIMEOUT);
+}
+
+void s5p_mipi_dsi_set_lpdr_timeout(struct mipi_dsim_device *dsim,
+ unsigned int timeout)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_TIMEOUT)) &
+ ~(0xffff << DSIM_LPDR_TOUT_SHIFT);
+
+ reg |= (timeout << DSIM_LPDR_TOUT_SHIFT);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_TIMEOUT);
+}
+
+void s5p_mipi_dsi_set_cpu_transfer_mode(struct mipi_dsim_device *dsim,
+ unsigned int lp)
+{
+ unsigned int reg = readl(dsim->reg_base + S5P_DSIM_ESCMODE);
+
+ reg &= ~DSIM_CMD_LPDT_LP;
+
+ reg |= lp << DSIM_CMD_LPDT_SHIFT;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_ESCMODE);
+}
+
+void s5p_mipi_dsi_set_lcdc_transfer_mode(struct mipi_dsim_device *dsim,
+ unsigned int lp)
+{
+ unsigned int reg = readl(dsim->reg_base + S5P_DSIM_ESCMODE);
+
+ reg &= ~DSIM_TX_LPDT_LP;
+
+ reg |= lp << DSIM_TX_LPDT_SHIFT;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_ESCMODE);
+}
+
+void s5p_mipi_dsi_enable_hs_clock(struct mipi_dsim_device *dsim,
+ unsigned int enable)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_CLKCTRL)) &
+ ~(1 << DSIM_TX_REQUEST_HSCLK_SHIFT);
+
+ reg |= enable << DSIM_TX_REQUEST_HSCLK_SHIFT;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_CLKCTRL);
+}
+
+void s5p_mipi_dsi_dp_dn_swap(struct mipi_dsim_device *dsim,
+ unsigned int swap_en)
+{
+ unsigned int reg = readl(dsim->reg_base + S5P_DSIM_PHYACCHR1);
+
+ reg &= ~(0x3 << 0);
+ reg |= (swap_en & 0x3) << 0;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_PHYACCHR1);
+}
+
+void s5p_mipi_dsi_hs_zero_ctrl(struct mipi_dsim_device *dsim,
+ unsigned int hs_zero)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_PLLCTRL)) &
+ ~(0xf << 28);
+
+ reg |= ((hs_zero & 0xf) << 28);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_PLLCTRL);
+}
+
+void s5p_mipi_dsi_prep_ctrl(struct mipi_dsim_device *dsim, unsigned int prep)
+{
+ unsigned int reg = (readl(dsim->reg_base + S5P_DSIM_PLLCTRL)) &
+ ~(0x7 << 20);
+
+ reg |= ((prep & 0x7) << 20);
+
+ writel(reg, dsim->reg_base + S5P_DSIM_PLLCTRL);
+}
+
+void s5p_mipi_dsi_clear_interrupt(struct mipi_dsim_device *dsim,
+ unsigned int int_src)
+{
+ writel(int_src, dsim->reg_base + S5P_DSIM_INTSRC);
+}
+
+void s5p_mipi_dsi_clear_all_interrupt(struct mipi_dsim_device *dsim)
+{
+ unsigned int reg = readl(dsim->reg_base + S5P_DSIM_INTSRC);
+
+ reg |= 0xffffffff;
+
+ writel(reg, dsim->reg_base + S5P_DSIM_INTSRC);
+}
+
+unsigned int s5p_mipi_dsi_is_pll_stable(struct mipi_dsim_device *dsim)
+{
+ unsigned int reg;
+
+ reg = readl(dsim->reg_base + S5P_DSIM_STATUS);
+
+ return reg & (1 << 31) ? 1 : 0;
+}
+
+unsigned int s5p_mipi_dsi_get_fifo_state(struct mipi_dsim_device *dsim)
+{
+ unsigned int ret;
+
+ ret = readl(dsim->reg_base + S5P_DSIM_FIFOCTRL) & ~(0x1f);
+
+ return ret;
+}
+
+void s5p_mipi_dsi_wr_tx_header(struct mipi_dsim_device *dsim,
+ unsigned int di, unsigned int data0, unsigned int data1)
+{
+ unsigned int reg = (data1 << 16) | (data0 << 8) | ((di & 0x3f) << 0);
+ writel(reg, dsim->reg_base + S5P_DSIM_PKTHDR);
+}
+
+unsigned int _s5p_mipi_dsi_get_frame_done_status(struct mipi_dsim_device *dsim)
+{
+ unsigned int reg = readl(dsim->reg_base + S5P_DSIM_INTSRC);
+
+ return (reg & INTSRC_FRAME_DONE) ? 1 : 0;
+}
+
+void _s5p_mipi_dsi_clear_frame_done(struct mipi_dsim_device *dsim)
+{
+ unsigned int reg = readl(dsim->reg_base + S5P_DSIM_INTSRC);
+
+ writel(reg | INTSRC_FRAME_DONE, dsim->reg_base +
+ S5P_DSIM_INTSRC);
+}
+
+void s5p_mipi_dsi_wr_tx_data(struct mipi_dsim_device *dsim,
+ unsigned int tx_data)
+{
+ writel(tx_data, dsim->reg_base + S5P_DSIM_PAYLOAD);
+}
+
+unsigned int s5p_mipi_dsi_get_int_status(struct mipi_dsim_device *dsim)
+{
+ return readl(dsim->reg_base + S5P_DSIM_INTSRC);
+}
+
+void s5p_mipi_dsi_clear_int_status(struct mipi_dsim_device *dsim,
+ unsigned int intSrc)
+{
+ writel(intSrc, dsim->reg_base + S5P_DSIM_INTSRC);
+}
+
+unsigned int s5p_mipi_dsi_get_FIFOCTRL_status(struct mipi_dsim_device *dsim)
+{
+ return readl(dsim->reg_base + S5P_DSIM_FIFOCTRL);
+}
--- /dev/null
+/* linux/drivers/video/s5p_mipi_dsi_lowlevel.h
+ *
+ * Header file for Samsung MIPI-DSI lowlevel driver.
+ *
+ * Copyright (c) 2011 Samsung Electronics
+ * InKi Dae <inki.dae@samsung.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#ifndef _S5P_MIPI_DSI_LOWLEVEL_H
+#define _S5P_MIPI_DSI_LOWLEVEL_H
+
+void s5p_mipi_dsi_func_reset(struct mipi_dsim_device *dsim);
+void s5p_mipi_dsi_sw_reset(struct mipi_dsim_device *dsim);
+void s5p_mipi_dsi_set_interrupt_mask(struct mipi_dsim_device *dsim,
+ unsigned int mode, unsigned int mask);
+void s5p_mipi_dsi_set_data_lane_number(struct mipi_dsim_device *dsim,
+ unsigned int count);
+void s5p_mipi_dsi_init_fifo_pointer(struct mipi_dsim_device *dsim,
+ unsigned int cfg);
+void s5p_mipi_dsi_set_phy_tunning(struct mipi_dsim_device *dsim,
+ unsigned int value);
+void s5p_mipi_dsi_set_phy_tunning(struct mipi_dsim_device *dsim,
+ unsigned int value);
+void s5p_mipi_dsi_set_main_disp_resol(struct mipi_dsim_device *dsim,
+ unsigned int vert_resol, unsigned int hori_resol);
+void s5p_mipi_dsi_set_main_disp_vporch(struct mipi_dsim_device *dsim,
+ unsigned int cmd_allow, unsigned int vfront, unsigned int vback);
+void s5p_mipi_dsi_set_main_disp_hporch(struct mipi_dsim_device *dsim,
+ unsigned int front, unsigned int back);
+void s5p_mipi_dsi_set_main_disp_sync_area(struct mipi_dsim_device *dsim,
+ unsigned int vert, unsigned int hori);
+void s5p_mipi_dsi_set_sub_disp_resol(struct mipi_dsim_device *dsim,
+ unsigned int vert, unsigned int hori);
+void s5p_mipi_dsi_init_config(struct mipi_dsim_device *dsim);
+void s5p_mipi_dsi_display_config(struct mipi_dsim_device *dsim);
+void s5p_mipi_dsi_set_data_lane_number(struct mipi_dsim_device *dsim,
+ unsigned int count);
+void s5p_mipi_dsi_enable_lane(struct mipi_dsim_device *dsim,
+ unsigned int lane, unsigned int enable);
+void s5p_mipi_dsi_enable_afc(struct mipi_dsim_device *dsim,
+ unsigned int enable, unsigned int afc_code);
+void s5p_mipi_dsi_enable_pll_bypass(struct mipi_dsim_device *dsim,
+ unsigned int enable);
+void s5p_mipi_dsi_set_pll_pms(struct mipi_dsim_device *dsim,
+ unsigned int p, unsigned int m, unsigned int s);
+void s5p_mipi_dsi_pll_freq_band(struct mipi_dsim_device *dsim,
+ unsigned int freq_band);
+void s5p_mipi_dsi_pll_freq(struct mipi_dsim_device *dsim,
+ unsigned int pre_divider, unsigned int main_divider,
+ unsigned int scaler);
+void s5p_mipi_dsi_pll_stable_time(struct mipi_dsim_device *dsim,
+ unsigned int lock_time);
+void s5p_mipi_dsi_enable_pll(struct mipi_dsim_device *dsim,
+ unsigned int enable);
+void s5p_mipi_dsi_set_byte_clock_src(struct mipi_dsim_device *dsim,
+ unsigned int src);
+void s5p_mipi_dsi_enable_byte_clock(struct mipi_dsim_device *dsim,
+ unsigned int enable);
+void s5p_mipi_dsi_set_esc_clk_prs(struct mipi_dsim_device *dsim,
+ unsigned int enable, unsigned int prs_val);
+void s5p_mipi_dsi_enable_esc_clk_on_lane(struct mipi_dsim_device *dsim,
+ unsigned int lane_sel, unsigned int enable);
+void s5p_mipi_dsi_force_dphy_stop_state(struct mipi_dsim_device *dsim,
+ unsigned int enable);
+unsigned int s5p_mipi_dsi_is_lane_state(struct mipi_dsim_device *dsim);
+void s5p_mipi_dsi_set_stop_state_counter(struct mipi_dsim_device *dsim,
+ unsigned int cnt_val);
+void s5p_mipi_dsi_set_bta_timeout(struct mipi_dsim_device *dsim,
+ unsigned int timeout);
+void s5p_mipi_dsi_set_lpdr_timeout(struct mipi_dsim_device *dsim,
+ unsigned int timeout);
+void s5p_mipi_dsi_set_lcdc_transfer_mode(struct mipi_dsim_device *dsim,
+ unsigned int lp);
+void s5p_mipi_dsi_set_cpu_transfer_mode(struct mipi_dsim_device *dsim,
+ unsigned int lp);
+void s5p_mipi_dsi_enable_hs_clock(struct mipi_dsim_device *dsim,
+ unsigned int enable);
+void s5p_mipi_dsi_dp_dn_swap(struct mipi_dsim_device *dsim,
+ unsigned int swap_en);
+void s5p_mipi_dsi_hs_zero_ctrl(struct mipi_dsim_device *dsim,
+ unsigned int hs_zero);
+void s5p_mipi_dsi_prep_ctrl(struct mipi_dsim_device *dsim, unsigned int
+ prep);
+void s5p_mipi_dsi_clear_interrupt(struct mipi_dsim_device *dsim,
+ unsigned int int_src);
+void s5p_mipi_dsi_clear_all_interrupt(struct mipi_dsim_device *dsim);
+unsigned int s5p_mipi_dsi_is_pll_stable(struct mipi_dsim_device *dsim);
+unsigned int s5p_mipi_dsi_get_fifo_state(struct mipi_dsim_device *dsim);
+unsigned int _s5p_mipi_dsi_get_frame_done_status(struct mipi_dsim_device *dsim);
+void _s5p_mipi_dsi_clear_frame_done(struct mipi_dsim_device *dsim);
+void s5p_mipi_dsi_wr_tx_header(struct mipi_dsim_device *dsim,
+ unsigned int di, unsigned int data0, unsigned int data1);
+void s5p_mipi_dsi_wr_tx_data(struct mipi_dsim_device *dsim,
+ unsigned int tx_data);
+unsigned int s5p_mipi_dsi_get_int_status(struct mipi_dsim_device *dsim);
+void s5p_mipi_dsi_clear_int_status(struct mipi_dsim_device *dsim,
+ unsigned int intSrc);
+unsigned int s5p_mipi_dsi_get_FIFOCTRL_status(struct mipi_dsim_device *dsim);
+
+#endif /* _S5P_MIPI_DSI_LOWLEVEL_H */
#ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT
/*
- * These two functions are only for dma allocator.
+ * These three functions are only for dma allocator.
* Don't use them in device drivers.
*/
int dma_alloc_from_coherent(struct device *dev, ssize_t size,
dma_addr_t *dma_handle, void **ret);
int dma_release_from_coherent(struct device *dev, int order, void *vaddr);
+int dma_mmap_from_coherent(struct device *dev, struct vm_area_struct *vma,
+ void *cpu_addr, size_t size, int *ret);
/*
* Standard interface
*/
--- /dev/null
+#ifndef ASM_DMA_CONTIGUOUS_H
+#define ASM_DMA_CONTIGUOUS_H
+
+#ifdef __KERNEL__
+#ifdef CONFIG_CMA
+
+#include <linux/device.h>
+#include <linux/dma-contiguous.h>
+
+static inline struct cma *dev_get_cma_area(struct device *dev)
+{
+ if (dev && dev->cma_area)
+ return dev->cma_area;
+ return dma_contiguous_default_area;
+}
+
+static inline void dev_set_cma_area(struct device *dev, struct cma *cma)
+{
+ if (dev)
+ dev->cma_area = cma;
+ if (!dev || !dma_contiguous_default_area)
+ dma_contiguous_default_area = cma;
+}
+
+#endif
+#endif
+
+#endif
#define dma_map_sg(d, s, n, r) dma_map_sg_attrs(d, s, n, r, NULL)
#define dma_unmap_sg(d, s, n, r) dma_unmap_sg_attrs(d, s, n, r, NULL)
+int
+dma_common_get_sgtable(struct device *dev, struct sg_table *sgt,
+ void *cpu_addr, dma_addr_t dma_addr, size_t size);
+
+static inline int
+dma_get_sgtable_attrs(struct device *dev, struct sg_table *sgt, void *cpu_addr,
+ dma_addr_t dma_addr, size_t size, struct dma_attrs *attrs)
+{
+ struct dma_map_ops *ops = get_dma_ops(dev);
+ BUG_ON(!ops);
+ if (ops->get_sgtable)
+ return ops->get_sgtable(dev, sgt, cpu_addr, dma_addr, size,
+ attrs);
+ return dma_common_get_sgtable(dev, sgt, cpu_addr, dma_addr, size);
+}
+
+#define dma_get_sgtable(d, t, v, h, s) dma_get_sgtable_attrs(d, t, v, h, s, NULL)
+
#endif
u32 height_mm;
};
+enum disp_panel_type {
+ MIPI_LCD,
+ DP_LCD
+};
+
/**
* Platform Specific Structure for DRM based FIMD.
*
u32 vidcon1;
unsigned int default_win;
unsigned int bpp;
+ unsigned int clock_rate;
+ enum disp_panel_type panel_type;
};
/**
struct dma_coherent_mem *dma_mem; /* internal for coherent mem
override */
+#ifdef CONFIG_CMA
+ struct cma *cma_area; /* contiguous memory area for dma
+ allocations */
+#endif
/* arch specific additions */
struct dev_archdata archdata;
DMA_ATTR_WEAK_ORDERING,
DMA_ATTR_WRITE_COMBINE,
DMA_ATTR_NON_CONSISTENT,
+ DMA_ATTR_NO_KERNEL_MAPPING,
+ DMA_ATTR_SKIP_CPU_SYNC,
DMA_ATTR_MAX,
};
* This Callback must not sleep.
* @kmap: maps a page from the buffer into kernel address space.
* @kunmap: [optional] unmaps a page from the buffer.
+ * @mmap: used to expose the backing storage to userspace. Note that the
+ * mapping needs to be coherent - if the exporter doesn't directly
+ * support this, it needs to fake coherency by shooting down any ptes
+ * when transitioning away from the cpu domain.
+ * @vmap: [optional] creates a virtual mapping for the buffer into kernel
+ * address space. Same restrictions as for vmap and friends apply.
+ * @vunmap: [optional] unmaps a vmap from the buffer
*/
struct dma_buf_ops {
int (*attach)(struct dma_buf *, struct device *,
void (*kunmap_atomic)(struct dma_buf *, unsigned long, void *);
void *(*kmap)(struct dma_buf *, unsigned long);
void (*kunmap)(struct dma_buf *, unsigned long, void *);
+
+ int (*mmap)(struct dma_buf *, struct vm_area_struct *vma);
+
+ void *(*vmap)(struct dma_buf *);
+ void (*vunmap)(struct dma_buf *, void *vaddr);
};
/**
void dma_buf_kunmap_atomic(struct dma_buf *, unsigned long, void *);
void *dma_buf_kmap(struct dma_buf *, unsigned long);
void dma_buf_kunmap(struct dma_buf *, unsigned long, void *);
+
+int dma_buf_mmap(struct dma_buf *, struct vm_area_struct *,
+ unsigned long);
+void *dma_buf_vmap(struct dma_buf *);
+void dma_buf_vunmap(struct dma_buf *, void *vaddr);
#else
static inline struct dma_buf_attachment *dma_buf_attach(struct dma_buf *dmabuf,
unsigned long pnum, void *vaddr)
{
}
+
+static inline int dma_buf_mmap(struct dma_buf *dmabuf,
+ struct vm_area_struct *vma,
+ unsigned long pgoff)
+{
+ return -ENODEV;
+}
+
+static inline void *dma_buf_vmap(struct dma_buf *dmabuf)
+{
+ return NULL;
+}
+
+static inline void dma_buf_vunmap(struct dma_buf *dmabuf, void *vaddr)
+{
+}
#endif /* CONFIG_DMA_SHARED_BUFFER */
#endif /* __DMA_BUF_H__ */
--- /dev/null
+#ifndef __LINUX_CMA_H
+#define __LINUX_CMA_H
+
+/*
+ * Contiguous Memory Allocator for DMA mapping framework
+ * Copyright (c) 2010-2011 by Samsung Electronics.
+ * Written by:
+ * Marek Szyprowski <m.szyprowski@samsung.com>
+ * Michal Nazarewicz <mina86@mina86.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of the
+ * License or (at your optional) any later version of the license.
+ */
+
+/*
+ * Contiguous Memory Allocator
+ *
+ * The Contiguous Memory Allocator (CMA) makes it possible to
+ * allocate big contiguous chunks of memory after the system has
+ * booted.
+ *
+ * Why is it needed?
+ *
+ * Various devices on embedded systems have no scatter-getter and/or
+ * IO map support and require contiguous blocks of memory to
+ * operate. They include devices such as cameras, hardware video
+ * coders, etc.
+ *
+ * Such devices often require big memory buffers (a full HD frame
+ * is, for instance, more then 2 mega pixels large, i.e. more than 6
+ * MB of memory), which makes mechanisms such as kmalloc() or
+ * alloc_page() ineffective.
+ *
+ * At the same time, a solution where a big memory region is
+ * reserved for a device is suboptimal since often more memory is
+ * reserved then strictly required and, moreover, the memory is
+ * inaccessible to page system even if device drivers don't use it.
+ *
+ * CMA tries to solve this issue by operating on memory regions
+ * where only movable pages can be allocated from. This way, kernel
+ * can use the memory for pagecache and when device driver requests
+ * it, allocated pages can be migrated.
+ *
+ * Driver usage
+ *
+ * CMA should not be used by the device drivers directly. It is
+ * only a helper framework for dma-mapping subsystem.
+ *
+ * For more information, see kernel-docs in drivers/base/dma-contiguous.c
+ */
+
+#ifdef __KERNEL__
+
+struct cma;
+struct page;
+struct device;
+
+#ifdef CONFIG_CMA
+
+/*
+ * There is always at least global CMA area and a few optional device
+ * private areas configured in kernel .config.
+ */
+#define MAX_CMA_AREAS (1 + CONFIG_CMA_AREAS)
+
+extern struct cma *dma_contiguous_default_area;
+
+void dma_contiguous_reserve(phys_addr_t addr_limit);
+int dma_declare_contiguous(struct device *dev, unsigned long size,
+ phys_addr_t base, phys_addr_t limit);
+
+struct page *dma_alloc_from_contiguous(struct device *dev, int count,
+ unsigned int order);
+bool dma_release_from_contiguous(struct device *dev, struct page *pages,
+ int count);
+
+#else
+
+#define MAX_CMA_AREAS (0)
+
+static inline void dma_contiguous_reserve(phys_addr_t limit) { }
+
+static inline
+int dma_declare_contiguous(struct device *dev, unsigned long size,
+ phys_addr_t base, phys_addr_t limit)
+{
+ return -ENOSYS;
+}
+
+static inline
+struct page *dma_alloc_from_contiguous(struct device *dev, int count,
+ unsigned int order)
+{
+ return NULL;
+}
+
+static inline
+bool dma_release_from_contiguous(struct device *dev, struct page *pages,
+ int count)
+{
+ return false;
+}
+
+#endif
+
+#endif
+
+#endif
int (*mmap)(struct device *, struct vm_area_struct *,
void *, dma_addr_t, size_t, struct dma_attrs *attrs);
+ int (*get_sgtable)(struct device *dev, struct sg_table *sgt, void *,
+ dma_addr_t, size_t, struct dma_attrs *attrs);
+
dma_addr_t (*map_page)(struct device *dev, struct page *page,
unsigned long offset, size_t size,
enum dma_data_direction dir,
}
#endif /* CONFIG_PM_SLEEP */
+#ifdef CONFIG_CMA
+
+/* The below functions must be run on a range from a single zone. */
+extern int alloc_contig_range(unsigned long start, unsigned long end,
+ unsigned migratetype);
+extern void free_contig_range(unsigned long pfn, unsigned nr_pages);
+
+/* CMA stuff */
+extern void init_cma_reserved_pageblock(struct page *page);
+
+#endif
+
#endif /* __LINUX_GFP_H */
*/
#define PAGE_ALLOC_COSTLY_ORDER 3
-#define MIGRATE_UNMOVABLE 0
-#define MIGRATE_RECLAIMABLE 1
-#define MIGRATE_MOVABLE 2
-#define MIGRATE_PCPTYPES 3 /* the number of types on the pcp lists */
-#define MIGRATE_RESERVE 3
-#define MIGRATE_ISOLATE 4 /* can't allocate from here */
-#define MIGRATE_TYPES 5
+enum {
+ MIGRATE_UNMOVABLE,
+ MIGRATE_RECLAIMABLE,
+ MIGRATE_MOVABLE,
+ MIGRATE_PCPTYPES, /* the number of types on the pcp lists */
+ MIGRATE_RESERVE = MIGRATE_PCPTYPES,
+#ifdef CONFIG_CMA
+ /*
+ * MIGRATE_CMA migration type is designed to mimic the way
+ * ZONE_MOVABLE works. Only movable pages can be allocated
+ * from MIGRATE_CMA pageblocks and page allocator never
+ * implicitly change migration type of MIGRATE_CMA pageblock.
+ *
+ * The way to use it is to change migratetype of a range of
+ * pageblocks to MIGRATE_CMA which can be done by
+ * __free_pageblock_cma() function. What is important though
+ * is that a range of pageblocks must be aligned to
+ * MAX_ORDER_NR_PAGES should biggest page be bigger then
+ * a single pageblock.
+ */
+ MIGRATE_CMA,
+#endif
+ MIGRATE_ISOLATE, /* can't allocate from here */
+ MIGRATE_TYPES
+};
+
+#ifdef CONFIG_CMA
+# define is_migrate_cma(migratetype) unlikely((migratetype) == MIGRATE_CMA)
+# define cma_wmark_pages(zone) zone->min_cma_pages
+#else
+# define is_migrate_cma(migratetype) false
+# define cma_wmark_pages(zone) 0
+#endif
#define for_each_migratetype_order(order, type) \
for (order = 0; order < MAX_ORDER; order++) \
#ifdef CONFIG_MEMORY_HOTPLUG
/* see spanned/present_pages for more description */
seqlock_t span_seqlock;
+#endif
+#ifdef CONFIG_CMA
+ /*
+ * CMA needs to increase watermark levels during the allocation
+ * process to make sure that the system is not starved.
+ */
+ unsigned long min_cma_pages;
#endif
struct free_area free_area[MAX_ORDER];
/*
* Changes migrate type in [start_pfn, end_pfn) to be MIGRATE_ISOLATE.
- * If specified range includes migrate types other than MOVABLE,
+ * If specified range includes migrate types other than MOVABLE or CMA,
* this will fail with -EBUSY.
*
* For isolating all pages in the range finally, the caller have to
* test it.
*/
extern int
-start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn);
+start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
+ unsigned migratetype);
/*
* Changes MIGRATE_ISOLATE to MIGRATE_MOVABLE.
* target range is [start_pfn, end_pfn)
*/
extern int
-undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn);
+undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
+ unsigned migratetype);
/*
- * test all pages in [start_pfn, end_pfn)are isolated or not.
+ * Test all pages in [start_pfn, end_pfn) are isolated or not.
*/
-extern int
-test_pages_isolated(unsigned long start_pfn, unsigned long end_pfn);
+int test_pages_isolated(unsigned long start_pfn, unsigned long end_pfn);
/*
- * Internal funcs.Changes pageblock's migrate type.
- * Please use make_pagetype_isolated()/make_pagetype_movable().
+ * Internal functions. Changes pageblock's migrate type.
*/
extern int set_migratetype_isolate(struct page *page);
-extern void unset_migratetype_isolate(struct page *page);
+extern void unset_migratetype_isolate(struct page *page, unsigned migratetype);
#endif
int __sg_alloc_table(struct sg_table *, unsigned int, unsigned int, gfp_t,
sg_alloc_fn *);
int sg_alloc_table(struct sg_table *, unsigned int, gfp_t);
+int sg_alloc_table_from_pages(struct sg_table *sgt,
+ struct page **pages, unsigned int n_pages,
+ unsigned long offset, unsigned long size,
+ gfp_t gfp_mask);
size_t sg_copy_from_buffer(struct scatterlist *sgl, unsigned int nents,
void *buf, size_t buflen);
V4L2_MEMORY_MMAP = 1,
V4L2_MEMORY_USERPTR = 2,
V4L2_MEMORY_OVERLAY = 3,
+ V4L2_MEMORY_DMABUF = 4,
};
/* see also http://vektor.theorem.ca/graphics/ycbcr/ */
* should be passed to mmap() called on the video node)
* @userptr: when memory is V4L2_MEMORY_USERPTR, a userspace pointer
* pointing to this plane
+ * @fd: when memory is V4L2_MEMORY_DMABUF, a userspace file
+ * descriptor associated with this plane
* @data_offset: offset in the plane to the start of data; usually 0,
* unless there is a header in front of the data
*
union {
__u32 mem_offset;
unsigned long userptr;
+ int fd;
} m;
__u32 data_offset;
__u32 reserved[11];
* (or a "cookie" that should be passed to mmap() as offset)
* @userptr: for non-multiplanar buffers with memory == V4L2_MEMORY_USERPTR;
* a userspace pointer pointing to this buffer
+ * @fd: for non-multiplanar buffers with memory == V4L2_MEMORY_DMABUF;
+ * a userspace file descriptor associated with this buffer
* @planes: for multiplanar buffers; userspace pointer to the array of plane
* info structs for this buffer
* @length: size in bytes of the buffer (NOT its payload) for single-plane
__u32 offset;
unsigned long userptr;
struct v4l2_plane *planes;
+ int fd;
} m;
__u32 length;
__u32 input;
#define V4L2_BUF_FLAG_NO_CACHE_INVALIDATE 0x0800
#define V4L2_BUF_FLAG_NO_CACHE_CLEAN 0x1000
+/**
+ * struct v4l2_exportbuffer - export of video buffer as DMABUF file descriptor
+ *
+ * @fd: file descriptor associated with DMABUF (set by driver)
+ * @mem_offset: buffer memory offset as returned by VIDIOC_QUERYBUF in struct
+ * v4l2_buffer::m.offset (for single-plane formats) or
+ * v4l2_plane::m.offset (for multi-planar formats)
+ * @flags: flags for newly created file, currently only O_CLOEXEC is
+ * supported, refer to manual of open syscall for more details
+ *
+ * Contains data used for exporting a video buffer as DMABUF file descriptor.
+ * The buffer is identified by a 'cookie' returned by VIDIOC_QUERYBUF
+ * (identical to the cookie used to mmap() the buffer to userspace). All
+ * reserved fields must be set to zero. The field reserved0 is expected to
+ * become a structure 'type' allowing an alternative layout of the structure
+ * content. Therefore this field should not be used for any other extensions.
+ */
+struct v4l2_exportbuffer {
+ __u32 fd;
+ __u32 reserved0;
+ __u32 mem_offset;
+ __u32 flags;
+ __u32 reserved[12];
+};
+
/*
* O V E R L A Y P R E V I E W
*/
#define VIDIOC_S_FBUF _IOW('V', 11, struct v4l2_framebuffer)
#define VIDIOC_OVERLAY _IOW('V', 14, int)
#define VIDIOC_QBUF _IOWR('V', 15, struct v4l2_buffer)
+#define VIDIOC_EXPBUF _IOWR('V', 16, struct v4l2_exportbuffer)
#define VIDIOC_DQBUF _IOWR('V', 17, struct v4l2_buffer)
#define VIDIOC_STREAMON _IOW('V', 18, int)
#define VIDIOC_STREAMOFF _IOW('V', 19, int)
#define VM_USERMAP 0x00000008 /* suitable for remap_vmalloc_range */
#define VM_VPAGES 0x00000010 /* buffer for pages was vmalloc'ed */
#define VM_UNLIST 0x00000020 /* vm_struct is not listed in vmlist */
+#define VM_DMA 0x00000040 /* used by dma-mapping framework */
/* bits [20..32] reserved for arch specific ioremap internals */
/*
struct page **pages;
unsigned int nr_pages;
phys_addr_t phys_addr;
- void *caller;
+ const void *caller;
};
/*
extern void *__vmalloc(unsigned long size, gfp_t gfp_mask, pgprot_t prot);
extern void *__vmalloc_node_range(unsigned long size, unsigned long align,
unsigned long start, unsigned long end, gfp_t gfp_mask,
- pgprot_t prot, int node, void *caller);
+ pgprot_t prot, int node, const void *caller);
extern void vfree(const void *addr);
extern void *vmap(struct page **pages, unsigned int count,
extern struct vm_struct *get_vm_area(unsigned long size, unsigned long flags);
extern struct vm_struct *get_vm_area_caller(unsigned long size,
- unsigned long flags, void *caller);
+ unsigned long flags, const void *caller);
extern struct vm_struct *__get_vm_area(unsigned long size, unsigned long flags,
unsigned long start, unsigned long end);
extern struct vm_struct *__get_vm_area_caller(unsigned long size,
unsigned long flags,
unsigned long start, unsigned long end,
- void *caller);
+ const void *caller);
extern struct vm_struct *remove_vm_area(const void *addr);
+extern struct vm_struct *find_vm_area(const void *addr);
extern int map_vm_area(struct vm_struct *area, pgprot_t prot,
struct page ***pages);
/* ioctl callbacks */
const struct v4l2_ioctl_ops *ioctl_ops;
+ DECLARE_BITMAP(valid_ioctls, BASE_VIDIOC_PRIVATE);
/* serialization lock */
+ DECLARE_BITMAP(dont_use_lock, BASE_VIDIOC_PRIVATE);
struct mutex *lock;
};
a dubious construction at best. */
void video_device_release_empty(struct video_device *vdev);
+/* returns true if cmd is a known V4L2 ioctl */
+bool v4l2_is_known_ioctl(unsigned int cmd);
+
+/* mark that this command shouldn't use core locking */
+static inline void v4l2_dont_use_lock(struct video_device *vdev, unsigned int cmd)
+{
+ if (_IOC_NR(cmd) < BASE_VIDIOC_PRIVATE)
+ set_bit(_IOC_NR(cmd), vdev->dont_use_lock);
+}
+
+/* Mark that this command isn't implemented, must be called before
+ video_device_register. See also the comments in determine_valid_ioctls().
+ This function allows drivers to provide just one v4l2_ioctl_ops struct, but
+ disable ioctls based on the specific card that is actually found. */
+static inline void v4l2_dont_use_cmd(struct video_device *vdev, unsigned int cmd)
+{
+ if (_IOC_NR(cmd) < BASE_VIDIOC_PRIVATE)
+ set_bit(_IOC_NR(cmd), vdev->valid_ioctls);
+}
+
/* helper functions to access driver private data. */
static inline void *video_get_drvdata(struct video_device *vdev)
{
int (*vidioc_reqbufs) (struct file *file, void *fh, struct v4l2_requestbuffers *b);
int (*vidioc_querybuf)(struct file *file, void *fh, struct v4l2_buffer *b);
int (*vidioc_qbuf) (struct file *file, void *fh, struct v4l2_buffer *b);
+ int (*vidioc_expbuf) (struct file *file, void *fh,
+ struct v4l2_exportbuffer *e);
int (*vidioc_dqbuf) (struct file *file, void *fh, struct v4l2_buffer *b);
int (*vidioc_create_bufs)(struct file *file, void *fh, struct v4l2_create_buffers *b);
int v4l2_m2m_querybuf(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
struct v4l2_buffer *buf);
+int v4l2_m2m_expbuf(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
+ struct v4l2_exportbuffer *eb);
+
int v4l2_m2m_qbuf(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
struct v4l2_buffer *buf);
int v4l2_m2m_dqbuf(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
#include <linux/mutex.h>
#include <linux/poll.h>
#include <linux/videodev2.h>
+#include <linux/dma-buf.h>
struct vb2_alloc_ctx;
struct vb2_fileio_data;
* argument to other ops in this structure
* @put_userptr: inform the allocator that a USERPTR buffer will no longer
* be used
+ * @attach_dmabuf: attach a shared struct dma_buf for a hardware operation;
+ * used for DMABUF memory types; alloc_ctx is the alloc context
+ * dbuf is the shared dma_buf; returns NULL on failure;
+ * allocator private per-buffer structure on success;
+ * this needs to be used for further accesses to the buffer
+ * @detach_dmabuf: inform the exporter of the buffer that the current DMABUF
+ * buffer is no longer used; the buf_priv argument is the
+ * allocator private per-buffer structure previously returned
+ * from the attach_dmabuf callback
+ * @map_dmabuf: request for access to the dmabuf from allocator; the allocator
+ * of dmabuf is informed that this driver is going to use the
+ * dmabuf
+ * @unmap_dmabuf: releases access control to the dmabuf - allocator is notified
+ * that this driver is done using the dmabuf for now
+ * @prepare: called everytime the buffer is passed from userspace to the
+ * driver, usefull for cache synchronisation, optional
+ * @finish: called everytime the buffer is passed back from the driver
+ * to the userspace, also optional
* @vaddr: return a kernel virtual address to a given memory buffer
* associated with the passed private structure or NULL if no
* such mapping exists
* Required ops for USERPTR types: get_userptr, put_userptr.
* Required ops for MMAP types: alloc, put, num_users, mmap.
* Required ops for read/write access types: alloc, put, num_users, vaddr
+ * Required ops for DMABUF types: attach_dmabuf, detach_dmabuf, map_dmabuf,
+ * unmap_dmabuf.
*/
struct vb2_mem_ops {
void *(*alloc)(void *alloc_ctx, unsigned long size);
void (*put)(void *buf_priv);
+ struct dma_buf *(*get_dmabuf)(void *buf_priv);
void *(*get_userptr)(void *alloc_ctx, unsigned long vaddr,
unsigned long size, int write);
void (*put_userptr)(void *buf_priv);
+ void (*prepare)(void *buf_priv);
+ void (*finish)(void *buf_priv);
+
+ void *(*attach_dmabuf)(void *alloc_ctx, struct dma_buf *dbuf,
+ unsigned long size, int write);
+ void (*detach_dmabuf)(void *buf_priv);
+ int (*map_dmabuf)(void *buf_priv);
+ void (*unmap_dmabuf)(void *buf_priv);
+
void *(*vaddr)(void *buf_priv);
void *(*cookie)(void *buf_priv);
struct vb2_plane {
void *mem_priv;
+ struct dma_buf *dbuf;
+ unsigned int dbuf_mapped;
};
/**
* @VB2_USERPTR: driver supports USERPTR with streaming API
* @VB2_READ: driver supports read() style access
* @VB2_WRITE: driver supports write() style access
+ * @VB2_DMABUF: driver supports DMABUF with streaming API
*/
enum vb2_io_modes {
VB2_MMAP = (1 << 0),
VB2_USERPTR = (1 << 1),
VB2_READ = (1 << 2),
VB2_WRITE = (1 << 3),
+ VB2_DMABUF = (1 << 4),
};
/**
void vb2_queue_release(struct vb2_queue *q);
int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b);
+int vb2_expbuf(struct vb2_queue *q, struct v4l2_exportbuffer *eb);
int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking);
int vb2_streamon(struct vb2_queue *q, enum v4l2_buf_type type);
int vb2_get_contig_userptr(unsigned long vaddr, unsigned long size,
struct vm_area_struct **res_vma, dma_addr_t *res_pa);
-int vb2_mmap_pfn_range(struct vm_area_struct *vma, unsigned long paddr,
- unsigned long size,
- const struct vm_operations_struct *vm_ops,
- void *priv);
-
struct vm_area_struct *vb2_get_vma(struct vm_area_struct *vma);
void vb2_put_vma(struct vm_area_struct *vma);
POWER_ALL
};
+enum link_training_type {
+ SW_LINK_TRAINING,
+ HW_LINK_TRAINING,
+};
+
struct video_info {
char *name;
struct exynos_dp_platdata {
struct video_info *video_info;
+ enum link_training_type training_type;
+
void (*phy_init)(void);
void (*phy_exit)(void);
};
}
EXPORT_SYMBOL(sg_alloc_table);
+/**
+ * sg_alloc_table_from_pages - Allocate and initialize an sg table from
+ * an array of pages
+ * @sgt: The sg table header to use
+ * @pages: Pointer to an array of page pointers
+ * @n_pages: Number of pages in the pages array
+ * @offset: Offset from a start of the first page to a start of a buffer
+ * @size: Number of valid bytes in the buffer (after offset)
+ * @gfp_mask: GFP allocation mask
+ *
+ * Description:
+ * Allocate and initialize an sg table from a list of pages. Continuous
+ * ranges of the pages are squashed into a single scatterlist node. A user
+ * may provide an offset at a start and a size of valid data in a buffer
+ * specified by the page array. The returned sg table is released by
+ * sg_free_table.
+ *
+ * Returns:
+ * 0 on success, negative error on failure
+ **/
+int sg_alloc_table_from_pages(struct sg_table *sgt,
+ struct page **pages, unsigned int n_pages,
+ unsigned long offset, unsigned long size,
+ gfp_t gfp_mask)
+{
+ unsigned int chunks;
+ unsigned int i;
+ unsigned int cur_page;
+ int ret;
+ struct scatterlist *s;
+
+ /* compute number of contiguous chunks */
+ chunks = 1;
+ for (i = 1; i < n_pages; ++i)
+ if (pages[i] != pages[i - 1] + 1)
+ ++chunks;
+
+ ret = sg_alloc_table(sgt, chunks, gfp_mask);
+ if (unlikely(ret))
+ return ret;
+
+ /* merging chunks and putting them into the scatterlist */
+ cur_page = 0;
+ for_each_sg(sgt->sgl, s, sgt->orig_nents, i) {
+ unsigned long chunk_size;
+ unsigned int j;
+
+ /* looking for the end of the current chunk */
+ for (j = cur_page + 1; j < n_pages; ++j)
+ if (pages[j] != pages[j - 1] + 1)
+ break;
+
+ chunk_size = ((j - cur_page) << PAGE_SHIFT) - offset;
+ sg_set_page(s, pages[cur_page], min(size, chunk_size), offset);
+ size -= chunk_size;
+ offset = 0;
+ cur_page = j;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(sg_alloc_table_from_pages);
+
/**
* sg_miter_start - start mapping iteration over a sg list
* @miter: sg mapping iter to be started
config MIGRATION
bool "Page migration"
def_bool y
- depends on NUMA || ARCH_ENABLE_MEMORY_HOTREMOVE || COMPACTION
+ depends on NUMA || ARCH_ENABLE_MEMORY_HOTREMOVE || COMPACTION || CMA
help
Allows the migration of the physical location of pages of processes
while the virtual addresses are not changed. This is useful in
readahead.o swap.o truncate.o vmscan.o shmem.o \
prio_tree.o util.o mmzone.o vmstat.o backing-dev.o \
page_isolation.o mm_init.o mmu_context.o percpu.o \
- $(mmu-y)
+ compaction.o $(mmu-y)
obj-y += init-mm.o
ifdef CONFIG_NO_BOOTMEM
obj-$(CONFIG_SPARSEMEM) += sparse.o
obj-$(CONFIG_SPARSEMEM_VMEMMAP) += sparse-vmemmap.o
obj-$(CONFIG_SLOB) += slob.o
-obj-$(CONFIG_COMPACTION) += compaction.o
obj-$(CONFIG_MMU_NOTIFIER) += mmu_notifier.o
obj-$(CONFIG_KSM) += ksm.o
obj-$(CONFIG_PAGE_POISONING) += debug-pagealloc.o
#include <linux/sysfs.h>
#include "internal.h"
+#if defined CONFIG_COMPACTION || defined CONFIG_CMA
+
#define CREATE_TRACE_POINTS
#include <trace/events/compaction.h>
-/*
- * compact_control is used to track pages being migrated and the free pages
- * they are being migrated to during memory compaction. The free_pfn starts
- * at the end of a zone and migrate_pfn begins at the start. Movable pages
- * are moved to the end of a zone during a compaction run and the run
- * completes when free_pfn <= migrate_pfn
- */
-struct compact_control {
- struct list_head freepages; /* List of free pages to migrate to */
- struct list_head migratepages; /* List of pages being migrated */
- unsigned long nr_freepages; /* Number of isolated free pages */
- unsigned long nr_migratepages; /* Number of pages to migrate */
- unsigned long free_pfn; /* isolate_freepages search base */
- unsigned long migrate_pfn; /* isolate_migratepages search base */
- bool sync; /* Synchronous migration */
-
- int order; /* order a direct compactor needs */
- int migratetype; /* MOVABLE, RECLAIMABLE etc */
- struct zone *zone;
-};
-
static unsigned long release_freepages(struct list_head *freelist)
{
struct page *page, *next;
return count;
}
-/* Isolate free pages onto a private freelist. Must hold zone->lock */
-static unsigned long isolate_freepages_block(struct zone *zone,
- unsigned long blockpfn,
- struct list_head *freelist)
+static void map_pages(struct list_head *list)
+{
+ struct page *page;
+
+ list_for_each_entry(page, list, lru) {
+ arch_alloc_page(page, 0);
+ kernel_map_pages(page, 1, 1);
+ }
+}
+
+static inline bool migrate_async_suitable(int migratetype)
+{
+ return is_migrate_cma(migratetype) || migratetype == MIGRATE_MOVABLE;
+}
+
+/*
+ * Isolate free pages onto a private freelist. Caller must hold zone->lock.
+ * If @strict is true, will abort returning 0 on any invalid PFNs or non-free
+ * pages inside of the pageblock (even though it may still end up isolating
+ * some pages).
+ */
+static unsigned long isolate_freepages_block(unsigned long blockpfn,
+ unsigned long end_pfn,
+ struct list_head *freelist,
+ bool strict)
{
- unsigned long zone_end_pfn, end_pfn;
int nr_scanned = 0, total_isolated = 0;
struct page *cursor;
- /* Get the last PFN we should scan for free pages at */
- zone_end_pfn = zone->zone_start_pfn + zone->spanned_pages;
- end_pfn = min(blockpfn + pageblock_nr_pages, zone_end_pfn);
-
- /* Find the first usable PFN in the block to initialse page cursor */
- for (; blockpfn < end_pfn; blockpfn++) {
- if (pfn_valid_within(blockpfn))
- break;
- }
cursor = pfn_to_page(blockpfn);
/* Isolate free pages. This assumes the block is valid */
int isolated, i;
struct page *page = cursor;
- if (!pfn_valid_within(blockpfn))
+ if (!pfn_valid_within(blockpfn)) {
+ if (strict)
+ return 0;
continue;
+ }
nr_scanned++;
- if (!PageBuddy(page))
+ if (!PageBuddy(page)) {
+ if (strict)
+ return 0;
continue;
+ }
/* Found a free page, break it into order-0 pages */
isolated = split_free_page(page);
+ if (!isolated && strict)
+ return 0;
total_isolated += isolated;
for (i = 0; i < isolated; i++) {
list_add(&page->lru, freelist);
return total_isolated;
}
-/* Returns true if the page is within a block suitable for migration to */
-static bool suitable_migration_target(struct page *page)
-{
-
- int migratetype = get_pageblock_migratetype(page);
-
- /* Don't interfere with memory hot-remove or the min_free_kbytes blocks */
- if (migratetype == MIGRATE_ISOLATE || migratetype == MIGRATE_RESERVE)
- return false;
-
- /* If the page is a large free page, then allow migration */
- if (PageBuddy(page) && page_order(page) >= pageblock_order)
- return true;
-
- /* If the block is MIGRATE_MOVABLE, allow migration */
- if (migratetype == MIGRATE_MOVABLE)
- return true;
-
- /* Otherwise skip the block */
- return false;
-}
-
-/*
- * Based on information in the current compact_control, find blocks
- * suitable for isolating free pages from and then isolate them.
+/**
+ * isolate_freepages_range() - isolate free pages.
+ * @start_pfn: The first PFN to start isolating.
+ * @end_pfn: The one-past-last PFN.
+ *
+ * Non-free pages, invalid PFNs, or zone boundaries within the
+ * [start_pfn, end_pfn) range are considered errors, cause function to
+ * undo its actions and return zero.
+ *
+ * Otherwise, function returns one-past-the-last PFN of isolated page
+ * (which may be greater then end_pfn if end fell in a middle of
+ * a free page).
*/
-static void isolate_freepages(struct zone *zone,
- struct compact_control *cc)
+unsigned long
+isolate_freepages_range(unsigned long start_pfn, unsigned long end_pfn)
{
- struct page *page;
- unsigned long high_pfn, low_pfn, pfn;
- unsigned long flags;
- int nr_freepages = cc->nr_freepages;
- struct list_head *freelist = &cc->freepages;
-
- /*
- * Initialise the free scanner. The starting point is where we last
- * scanned from (or the end of the zone if starting). The low point
- * is the end of the pageblock the migration scanner is using.
- */
- pfn = cc->free_pfn;
- low_pfn = cc->migrate_pfn + pageblock_nr_pages;
+ unsigned long isolated, pfn, block_end_pfn, flags;
+ struct zone *zone = NULL;
+ LIST_HEAD(freelist);
- /*
- * Take care that if the migration scanner is at the end of the zone
- * that the free scanner does not accidentally move to the next zone
- * in the next isolation cycle.
- */
- high_pfn = min(low_pfn, pfn);
-
- /*
- * Isolate free pages until enough are available to migrate the
- * pages on cc->migratepages. We stop searching if the migrate
- * and free page scanners meet or enough free pages are isolated.
- */
- for (; pfn > low_pfn && cc->nr_migratepages > nr_freepages;
- pfn -= pageblock_nr_pages) {
- unsigned long isolated;
+ if (pfn_valid(start_pfn))
+ zone = page_zone(pfn_to_page(start_pfn));
- if (!pfn_valid(pfn))
- continue;
+ for (pfn = start_pfn; pfn < end_pfn; pfn += isolated) {
+ if (!pfn_valid(pfn) || zone != page_zone(pfn_to_page(pfn)))
+ break;
/*
- * Check for overlapping nodes/zones. It's possible on some
- * configurations to have a setup like
- * node0 node1 node0
- * i.e. it's possible that all pages within a zones range of
- * pages do not belong to a single zone.
+ * On subsequent iterations ALIGN() is actually not needed,
+ * but we keep it that we not to complicate the code.
*/
- page = pfn_to_page(pfn);
- if (page_zone(page) != zone)
- continue;
+ block_end_pfn = ALIGN(pfn + 1, pageblock_nr_pages);
+ block_end_pfn = min(block_end_pfn, end_pfn);
- /* Check the block is suitable for migration */
- if (!suitable_migration_target(page))
- continue;
+ spin_lock_irqsave(&zone->lock, flags);
+ isolated = isolate_freepages_block(pfn, block_end_pfn,
+ &freelist, true);
+ spin_unlock_irqrestore(&zone->lock, flags);
/*
- * Found a block suitable for isolating free pages from. Now
- * we disabled interrupts, double check things are ok and
- * isolate the pages. This is to minimise the time IRQs
- * are disabled
+ * In strict mode, isolate_freepages_block() returns 0 if
+ * there are any holes in the block (ie. invalid PFNs or
+ * non-free pages).
*/
- isolated = 0;
- spin_lock_irqsave(&zone->lock, flags);
- if (suitable_migration_target(page)) {
- isolated = isolate_freepages_block(zone, pfn, freelist);
- nr_freepages += isolated;
- }
- spin_unlock_irqrestore(&zone->lock, flags);
+ if (!isolated)
+ break;
/*
- * Record the highest PFN we isolated pages from. When next
- * looking for free pages, the search will restart here as
- * page migration may have returned some pages to the allocator
+ * If we managed to isolate pages, it is always (1 << n) *
+ * pageblock_nr_pages for some non-negative n. (Max order
+ * page may span two pageblocks).
*/
- if (isolated)
- high_pfn = max(high_pfn, pfn);
}
/* split_free_page does not map the pages */
- list_for_each_entry(page, freelist, lru) {
- arch_alloc_page(page, 0);
- kernel_map_pages(page, 1, 1);
+ map_pages(&freelist);
+
+ if (pfn < end_pfn) {
+ /* Loop terminated early, cleanup. */
+ release_freepages(&freelist);
+ return 0;
}
- cc->free_pfn = high_pfn;
- cc->nr_freepages = nr_freepages;
+ /* We don't use freelists for anything. */
+ return pfn;
}
/* Update the number of anon and file isolated pages in the zone */
return isolated > (inactive + active) / 2;
}
-/* possible outcome of isolate_migratepages */
-typedef enum {
- ISOLATE_ABORT, /* Abort compaction now */
- ISOLATE_NONE, /* No pages isolated, continue scanning */
- ISOLATE_SUCCESS, /* Pages isolated, migrate */
-} isolate_migrate_t;
-
-/*
- * Isolate all pages that can be migrated from the block pointed to by
- * the migrate scanner within compact_control.
+/**
+ * isolate_migratepages_range() - isolate all migrate-able pages in range.
+ * @zone: Zone pages are in.
+ * @cc: Compaction control structure.
+ * @low_pfn: The first PFN of the range.
+ * @end_pfn: The one-past-the-last PFN of the range.
+ *
+ * Isolate all pages that can be migrated from the range specified by
+ * [low_pfn, end_pfn). Returns zero if there is a fatal signal
+ * pending), otherwise PFN of the first page that was not scanned
+ * (which may be both less, equal to or more then end_pfn).
+ *
+ * Assumes that cc->migratepages is empty and cc->nr_migratepages is
+ * zero.
+ *
+ * Apart from cc->migratepages and cc->nr_migratetypes this function
+ * does not modify any cc's fields, in particular it does not modify
+ * (or read for that matter) cc->migrate_pfn.
*/
-static isolate_migrate_t isolate_migratepages(struct zone *zone,
- struct compact_control *cc)
+unsigned long
+isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
+ unsigned long low_pfn, unsigned long end_pfn)
{
- unsigned long low_pfn, end_pfn;
unsigned long last_pageblock_nr = 0, pageblock_nr;
unsigned long nr_scanned = 0, nr_isolated = 0;
struct list_head *migratelist = &cc->migratepages;
isolate_mode_t mode = ISOLATE_ACTIVE|ISOLATE_INACTIVE;
- /* Do not scan outside zone boundaries */
- low_pfn = max(cc->migrate_pfn, zone->zone_start_pfn);
-
- /* Only scan within a pageblock boundary */
- end_pfn = ALIGN(low_pfn + pageblock_nr_pages, pageblock_nr_pages);
-
- /* Do not cross the free scanner or scan within a memory hole */
- if (end_pfn > cc->free_pfn || !pfn_valid(low_pfn)) {
- cc->migrate_pfn = end_pfn;
- return ISOLATE_NONE;
- }
-
/*
* Ensure that there are not too many pages isolated from the LRU
* list by either parallel reclaimers or compaction. If there are,
while (unlikely(too_many_isolated(zone))) {
/* async migration should just abort */
if (!cc->sync)
- return ISOLATE_ABORT;
+ return 0;
congestion_wait(BLK_RW_ASYNC, HZ/10);
if (fatal_signal_pending(current))
- return ISOLATE_ABORT;
+ return 0;
}
/* Time to isolate some pages for migration */
*/
pageblock_nr = low_pfn >> pageblock_order;
if (!cc->sync && last_pageblock_nr != pageblock_nr &&
- get_pageblock_migratetype(page) != MIGRATE_MOVABLE) {
+ !migrate_async_suitable(get_pageblock_migratetype(page))) {
low_pfn += pageblock_nr_pages;
low_pfn = ALIGN(low_pfn, pageblock_nr_pages) - 1;
last_pageblock_nr = pageblock_nr;
acct_isolated(zone, cc);
spin_unlock_irq(&zone->lru_lock);
- cc->migrate_pfn = low_pfn;
trace_mm_compaction_isolate_migratepages(nr_scanned, nr_isolated);
- return ISOLATE_SUCCESS;
+ return low_pfn;
+}
+
+#endif /* CONFIG_COMPACTION || CONFIG_CMA */
+#ifdef CONFIG_COMPACTION
+
+/* Returns true if the page is within a block suitable for migration to */
+static bool suitable_migration_target(struct page *page)
+{
+
+ int migratetype = get_pageblock_migratetype(page);
+
+ /* Don't interfere with memory hot-remove or the min_free_kbytes blocks */
+ if (migratetype == MIGRATE_ISOLATE || migratetype == MIGRATE_RESERVE)
+ return false;
+
+ /* If the page is a large free page, then allow migration */
+ if (PageBuddy(page) && page_order(page) >= pageblock_order)
+ return true;
+
+ /* If the block is MIGRATE_MOVABLE or MIGRATE_CMA, allow migration */
+ if (migrate_async_suitable(migratetype))
+ return true;
+
+ /* Otherwise skip the block */
+ return false;
+}
+
+/*
+ * Based on information in the current compact_control, find blocks
+ * suitable for isolating free pages from and then isolate them.
+ */
+static void isolate_freepages(struct zone *zone,
+ struct compact_control *cc)
+{
+ struct page *page;
+ unsigned long high_pfn, low_pfn, pfn, zone_end_pfn, end_pfn;
+ unsigned long flags;
+ int nr_freepages = cc->nr_freepages;
+ struct list_head *freelist = &cc->freepages;
+
+ /*
+ * Initialise the free scanner. The starting point is where we last
+ * scanned from (or the end of the zone if starting). The low point
+ * is the end of the pageblock the migration scanner is using.
+ */
+ pfn = cc->free_pfn;
+ low_pfn = cc->migrate_pfn + pageblock_nr_pages;
+
+ /*
+ * Take care that if the migration scanner is at the end of the zone
+ * that the free scanner does not accidentally move to the next zone
+ * in the next isolation cycle.
+ */
+ high_pfn = min(low_pfn, pfn);
+
+ zone_end_pfn = zone->zone_start_pfn + zone->spanned_pages;
+
+ /*
+ * Isolate free pages until enough are available to migrate the
+ * pages on cc->migratepages. We stop searching if the migrate
+ * and free page scanners meet or enough free pages are isolated.
+ */
+ for (; pfn > low_pfn && cc->nr_migratepages > nr_freepages;
+ pfn -= pageblock_nr_pages) {
+ unsigned long isolated;
+
+ if (!pfn_valid(pfn))
+ continue;
+
+ /*
+ * Check for overlapping nodes/zones. It's possible on some
+ * configurations to have a setup like
+ * node0 node1 node0
+ * i.e. it's possible that all pages within a zones range of
+ * pages do not belong to a single zone.
+ */
+ page = pfn_to_page(pfn);
+ if (page_zone(page) != zone)
+ continue;
+
+ /* Check the block is suitable for migration */
+ if (!suitable_migration_target(page))
+ continue;
+
+ /*
+ * Found a block suitable for isolating free pages from. Now
+ * we disabled interrupts, double check things are ok and
+ * isolate the pages. This is to minimise the time IRQs
+ * are disabled
+ */
+ isolated = 0;
+ spin_lock_irqsave(&zone->lock, flags);
+ if (suitable_migration_target(page)) {
+ end_pfn = min(pfn + pageblock_nr_pages, zone_end_pfn);
+ isolated = isolate_freepages_block(pfn, end_pfn,
+ freelist, false);
+ nr_freepages += isolated;
+ }
+ spin_unlock_irqrestore(&zone->lock, flags);
+
+ /*
+ * Record the highest PFN we isolated pages from. When next
+ * looking for free pages, the search will restart here as
+ * page migration may have returned some pages to the allocator
+ */
+ if (isolated)
+ high_pfn = max(high_pfn, pfn);
+ }
+
+ /* split_free_page does not map the pages */
+ map_pages(freelist);
+
+ cc->free_pfn = high_pfn;
+ cc->nr_freepages = nr_freepages;
}
/*
cc->nr_freepages = nr_freepages;
}
+/* possible outcome of isolate_migratepages */
+typedef enum {
+ ISOLATE_ABORT, /* Abort compaction now */
+ ISOLATE_NONE, /* No pages isolated, continue scanning */
+ ISOLATE_SUCCESS, /* Pages isolated, migrate */
+} isolate_migrate_t;
+
+/*
+ * Isolate all pages that can be migrated from the block pointed to by
+ * the migrate scanner within compact_control.
+ */
+static isolate_migrate_t isolate_migratepages(struct zone *zone,
+ struct compact_control *cc)
+{
+ unsigned long low_pfn, end_pfn;
+
+ /* Do not scan outside zone boundaries */
+ low_pfn = max(cc->migrate_pfn, zone->zone_start_pfn);
+
+ /* Only scan within a pageblock boundary */
+ end_pfn = ALIGN(low_pfn + pageblock_nr_pages, pageblock_nr_pages);
+
+ /* Do not cross the free scanner or scan within a memory hole */
+ if (end_pfn > cc->free_pfn || !pfn_valid(low_pfn)) {
+ cc->migrate_pfn = end_pfn;
+ return ISOLATE_NONE;
+ }
+
+ /* Perform the isolation */
+ low_pfn = isolate_migratepages_range(zone, cc, low_pfn, end_pfn);
+ if (!low_pfn)
+ return ISOLATE_ABORT;
+
+ cc->migrate_pfn = low_pfn;
+
+ return ISOLATE_SUCCESS;
+}
+
static int compact_finished(struct zone *zone,
struct compact_control *cc)
{
return device_remove_file(&node->dev, &dev_attr_compact);
}
#endif /* CONFIG_SYSFS && CONFIG_NUMA */
+
+#endif /* CONFIG_COMPACTION */
extern bool is_free_buddy_page(struct page *page);
#endif
+#if defined CONFIG_COMPACTION || defined CONFIG_CMA
+
+/*
+ * in mm/compaction.c
+ */
+/*
+ * compact_control is used to track pages being migrated and the free pages
+ * they are being migrated to during memory compaction. The free_pfn starts
+ * at the end of a zone and migrate_pfn begins at the start. Movable pages
+ * are moved to the end of a zone during a compaction run and the run
+ * completes when free_pfn <= migrate_pfn
+ */
+struct compact_control {
+ struct list_head freepages; /* List of free pages to migrate to */
+ struct list_head migratepages; /* List of pages being migrated */
+ unsigned long nr_freepages; /* Number of isolated free pages */
+ unsigned long nr_migratepages; /* Number of pages to migrate */
+ unsigned long free_pfn; /* isolate_freepages search base */
+ unsigned long migrate_pfn; /* isolate_migratepages search base */
+ bool sync; /* Synchronous migration */
+
+ int order; /* order a direct compactor needs */
+ int migratetype; /* MOVABLE, RECLAIMABLE etc */
+ struct zone *zone;
+};
+
+unsigned long
+isolate_freepages_range(unsigned long start_pfn, unsigned long end_pfn);
+unsigned long
+isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
+ unsigned long low_pfn, unsigned long end_pfn);
+
+#endif
/*
* function for dealing with page's order in buddy system.
/* Not a free page */
ret = 1;
}
- unset_migratetype_isolate(p);
+ unset_migratetype_isolate(p, MIGRATE_MOVABLE);
unlock_memory_hotplug();
return ret;
}
nr_pages = end_pfn - start_pfn;
/* set above range as isolated */
- ret = start_isolate_page_range(start_pfn, end_pfn);
+ ret = start_isolate_page_range(start_pfn, end_pfn, MIGRATE_MOVABLE);
if (ret)
goto out;
We cannot do rollback at this point. */
offline_isolated_pages(start_pfn, end_pfn);
/* reset pagetype flags and makes migrate type to be MOVABLE */
- undo_isolate_page_range(start_pfn, end_pfn);
+ undo_isolate_page_range(start_pfn, end_pfn, MIGRATE_MOVABLE);
/* removal success */
zone->present_pages -= offlined_pages;
zone->zone_pgdat->node_present_pages -= offlined_pages;
start_pfn, end_pfn);
memory_notify(MEM_CANCEL_OFFLINE, &arg);
/* pushback to free area */
- undo_isolate_page_range(start_pfn, end_pfn);
+ undo_isolate_page_range(start_pfn, end_pfn, MIGRATE_MOVABLE);
out:
unlock_memory_hotplug();
#include <linux/ftrace_event.h>
#include <linux/memcontrol.h>
#include <linux/prefetch.h>
+#include <linux/migrate.h>
#include <linux/page-debug-flags.h>
#include <asm/tlbflush.h>
* free pages of length of (1 << order) and marked with _mapcount -2. Page's
* order is recorded in page_private(page) field.
* So when we are allocating or freeing one, we can derive the state of the
- * other. That is, if we allocate a small block, and both were
- * free, the remainder of the region must be split into blocks.
+ * other. That is, if we allocate a small block, and both were
+ * free, the remainder of the region must be split into blocks.
* If a block is freed, and its buddy is also free, then this
- * triggers coalescing into a block of larger size.
+ * triggers coalescing into a block of larger size.
*
* -- wli
*/
__free_pages(page, order);
}
+#ifdef CONFIG_CMA
+/* Free whole pageblock and set it's migration type to MIGRATE_CMA. */
+void __init init_cma_reserved_pageblock(struct page *page)
+{
+ unsigned i = pageblock_nr_pages;
+ struct page *p = page;
+
+ do {
+ __ClearPageReserved(p);
+ set_page_count(p, 0);
+ } while (++p, --i);
+
+ set_page_refcounted(page);
+ set_pageblock_migratetype(page, MIGRATE_CMA);
+ __free_pages(page, pageblock_order);
+ totalram_pages += pageblock_nr_pages;
+}
+#endif
/*
* The order of subdivision here is critical for the IO subsystem.
* This array describes the order lists are fallen back to when
* the free lists for the desirable migrate type are depleted
*/
-static int fallbacks[MIGRATE_TYPES][MIGRATE_TYPES-1] = {
- [MIGRATE_UNMOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_RESERVE },
- [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_RESERVE },
- [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_RESERVE },
- [MIGRATE_RESERVE] = { MIGRATE_RESERVE, MIGRATE_RESERVE, MIGRATE_RESERVE }, /* Never used */
+static int fallbacks[MIGRATE_TYPES][4] = {
+ [MIGRATE_UNMOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_RESERVE },
+ [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_RESERVE },
+#ifdef CONFIG_CMA
+ [MIGRATE_MOVABLE] = { MIGRATE_CMA, MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_RESERVE },
+ [MIGRATE_CMA] = { MIGRATE_RESERVE }, /* Never used */
+#else
+ [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_RESERVE },
+#endif
+ [MIGRATE_RESERVE] = { MIGRATE_RESERVE }, /* Never used */
+ [MIGRATE_ISOLATE] = { MIGRATE_RESERVE }, /* Never used */
};
/*
/* Find the largest possible block of pages in the other list */
for (current_order = MAX_ORDER-1; current_order >= order;
--current_order) {
- for (i = 0; i < MIGRATE_TYPES - 1; i++) {
+ for (i = 0;; i++) {
migratetype = fallbacks[start_migratetype][i];
/* MIGRATE_RESERVE handled later if necessary */
if (migratetype == MIGRATE_RESERVE)
- continue;
+ break;
area = &(zone->free_area[current_order]);
if (list_empty(&area->free_list[migratetype]))
* pages to the preferred allocation list. If falling
* back for a reclaimable kernel allocation, be more
* aggressive about taking ownership of free pages
+ *
+ * On the other hand, never change migration
+ * type of MIGRATE_CMA pageblocks nor move CMA
+ * pages on different free lists. We don't
+ * want unmovable pages to be allocated from
+ * MIGRATE_CMA areas.
*/
- if (unlikely(current_order >= (pageblock_order >> 1)) ||
- start_migratetype == MIGRATE_RECLAIMABLE ||
- page_group_by_mobility_disabled) {
- unsigned long pages;
+ if (!is_migrate_cma(migratetype) &&
+ (unlikely(current_order >= pageblock_order / 2) ||
+ start_migratetype == MIGRATE_RECLAIMABLE ||
+ page_group_by_mobility_disabled)) {
+ int pages;
pages = move_freepages_block(zone, page,
start_migratetype);
rmv_page_order(page);
/* Take ownership for orders >= pageblock_order */
- if (current_order >= pageblock_order)
+ if (current_order >= pageblock_order &&
+ !is_migrate_cma(migratetype))
change_pageblock_range(page, current_order,
start_migratetype);
- expand(zone, page, order, current_order, area, migratetype);
+ expand(zone, page, order, current_order, area,
+ is_migrate_cma(migratetype)
+ ? migratetype : start_migratetype);
trace_mm_page_alloc_extfrag(page, order, current_order,
start_migratetype, migratetype);
return page;
}
-/*
+/*
* Obtain a specified number of elements from the buddy allocator, all under
* a single hold of the lock, for efficiency. Add them to the supplied list.
* Returns the number of new pages which were placed at *list.
*/
-static int rmqueue_bulk(struct zone *zone, unsigned int order,
+static int rmqueue_bulk(struct zone *zone, unsigned int order,
unsigned long count, struct list_head *list,
int migratetype, int cold)
{
- int i;
-
+ int mt = migratetype, i;
+
spin_lock(&zone->lock);
for (i = 0; i < count; ++i) {
struct page *page = __rmqueue(zone, order, migratetype);
list_add(&page->lru, list);
else
list_add_tail(&page->lru, list);
- set_page_private(page, migratetype);
+ if (IS_ENABLED(CONFIG_CMA)) {
+ mt = get_pageblock_migratetype(page);
+ if (!is_migrate_cma(mt) && mt != MIGRATE_ISOLATE)
+ mt = migratetype;
+ }
+ set_page_private(page, mt);
list = &page->lru;
}
__mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
if (order >= pageblock_order - 1) {
struct page *endpage = page + (1 << order) - 1;
- for (; page < endpage; page += pageblock_nr_pages)
- set_pageblock_migratetype(page, MIGRATE_MOVABLE);
+ for (; page < endpage; page += pageblock_nr_pages) {
+ int mt = get_pageblock_migratetype(page);
+ if (mt != MIGRATE_ISOLATE && !is_migrate_cma(mt))
+ set_pageblock_migratetype(page,
+ MIGRATE_MOVABLE);
+ }
}
return 1 << order;
}
#endif /* CONFIG_COMPACTION */
-/* The really slow allocator path where we enter direct reclaim */
-static inline struct page *
-__alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
- struct zonelist *zonelist, enum zone_type high_zoneidx,
- nodemask_t *nodemask, int alloc_flags, struct zone *preferred_zone,
- int migratetype, unsigned long *did_some_progress)
+/* Perform direct synchronous page reclaim */
+static int
+__perform_reclaim(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist,
+ nodemask_t *nodemask)
{
- struct page *page = NULL;
struct reclaim_state reclaim_state;
- bool drained = false;
+ int progress;
cond_resched();
reclaim_state.reclaimed_slab = 0;
current->reclaim_state = &reclaim_state;
- *did_some_progress = try_to_free_pages(zonelist, order, gfp_mask, nodemask);
+ progress = try_to_free_pages(zonelist, order, gfp_mask, nodemask);
current->reclaim_state = NULL;
lockdep_clear_current_reclaim_state();
cond_resched();
+ return progress;
+}
+
+/* The really slow allocator path where we enter direct reclaim */
+static inline struct page *
+__alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
+ struct zonelist *zonelist, enum zone_type high_zoneidx,
+ nodemask_t *nodemask, int alloc_flags, struct zone *preferred_zone,
+ int migratetype, unsigned long *did_some_progress)
+{
+ struct page *page = NULL;
+ bool drained = false;
+
+ *did_some_progress = __perform_reclaim(gfp_mask, order, zonelist,
+ nodemask);
if (unlikely(!(*did_some_progress)))
return NULL;
init_waitqueue_head(&pgdat->kswapd_wait);
pgdat->kswapd_max_order = 0;
pgdat_page_cgroup_init(pgdat);
-
+
for (j = 0; j < MAX_NR_ZONES; j++) {
struct zone *zone = pgdat->node_zones + j;
unsigned long size, realsize, memmap_pages;
calculate_totalreserve_pages();
}
-/**
- * setup_per_zone_wmarks - called when min_free_kbytes changes
- * or when memory is hot-{added|removed}
- *
- * Ensures that the watermark[min,low,high] values for each zone are set
- * correctly with respect to min_free_kbytes.
- */
-void setup_per_zone_wmarks(void)
+static void __setup_per_zone_wmarks(void)
{
unsigned long pages_min = min_free_kbytes >> (PAGE_SHIFT - 10);
unsigned long lowmem_pages = 0;
zone->watermark[WMARK_LOW] = min_wmark_pages(zone) + (tmp >> 2);
zone->watermark[WMARK_HIGH] = min_wmark_pages(zone) + (tmp >> 1);
+
+ zone->watermark[WMARK_MIN] += cma_wmark_pages(zone);
+ zone->watermark[WMARK_LOW] += cma_wmark_pages(zone);
+ zone->watermark[WMARK_HIGH] += cma_wmark_pages(zone);
+
setup_zone_migrate_reserve(zone);
spin_unlock_irqrestore(&zone->lock, flags);
}
calculate_totalreserve_pages();
}
+/**
+ * setup_per_zone_wmarks - called when min_free_kbytes changes
+ * or when memory is hot-{added|removed}
+ *
+ * Ensures that the watermark[min,low,high] values for each zone are set
+ * correctly with respect to min_free_kbytes.
+ */
+void setup_per_zone_wmarks(void)
+{
+ mutex_lock(&zonelists_mutex);
+ __setup_per_zone_wmarks();
+ mutex_unlock(&zonelists_mutex);
+}
+
/*
* The inactive anon list should be small enough that the VM never has to
* do too much work, but large enough that each inactive page has a chance
__count_immobile_pages(struct zone *zone, struct page *page, int count)
{
unsigned long pfn, iter, found;
+ int mt;
+
/*
* For avoiding noise data, lru_add_drain_all() should be called
* If ZONE_MOVABLE, the zone never contains immobile pages
*/
if (zone_idx(zone) == ZONE_MOVABLE)
return true;
-
- if (get_pageblock_migratetype(page) == MIGRATE_MOVABLE)
+ mt = get_pageblock_migratetype(page);
+ if (mt == MIGRATE_MOVABLE || is_migrate_cma(mt))
return true;
pfn = page_to_pfn(page);
return ret;
}
-void unset_migratetype_isolate(struct page *page)
+void unset_migratetype_isolate(struct page *page, unsigned migratetype)
{
struct zone *zone;
unsigned long flags;
spin_lock_irqsave(&zone->lock, flags);
if (get_pageblock_migratetype(page) != MIGRATE_ISOLATE)
goto out;
- set_pageblock_migratetype(page, MIGRATE_MOVABLE);
- move_freepages_block(zone, page, MIGRATE_MOVABLE);
+ set_pageblock_migratetype(page, migratetype);
+ move_freepages_block(zone, page, migratetype);
out:
spin_unlock_irqrestore(&zone->lock, flags);
}
+#ifdef CONFIG_CMA
+
+static unsigned long pfn_max_align_down(unsigned long pfn)
+{
+ return pfn & ~(max_t(unsigned long, MAX_ORDER_NR_PAGES,
+ pageblock_nr_pages) - 1);
+}
+
+static unsigned long pfn_max_align_up(unsigned long pfn)
+{
+ return ALIGN(pfn, max_t(unsigned long, MAX_ORDER_NR_PAGES,
+ pageblock_nr_pages));
+}
+
+static struct page *
+__alloc_contig_migrate_alloc(struct page *page, unsigned long private,
+ int **resultp)
+{
+ return alloc_page(GFP_HIGHUSER_MOVABLE);
+}
+
+/* [start, end) must belong to a single zone. */
+static int __alloc_contig_migrate_range(unsigned long start, unsigned long end)
+{
+ /* This function is based on compact_zone() from compaction.c. */
+
+ unsigned long pfn = start;
+ unsigned int tries = 0;
+ int ret = 0;
+
+ struct compact_control cc = {
+ .nr_migratepages = 0,
+ .order = -1,
+ .zone = page_zone(pfn_to_page(start)),
+ .sync = true,
+ };
+ INIT_LIST_HEAD(&cc.migratepages);
+
+ migrate_prep_local();
+
+ while (pfn < end || !list_empty(&cc.migratepages)) {
+ if (fatal_signal_pending(current)) {
+ ret = -EINTR;
+ break;
+ }
+
+ if (list_empty(&cc.migratepages)) {
+ cc.nr_migratepages = 0;
+ pfn = isolate_migratepages_range(cc.zone, &cc,
+ pfn, end);
+ if (!pfn) {
+ ret = -EINTR;
+ break;
+ }
+ tries = 0;
+ } else if (++tries == 5) {
+ ret = ret < 0 ? ret : -EBUSY;
+ break;
+ }
+
+ ret = migrate_pages(&cc.migratepages,
+ __alloc_contig_migrate_alloc,
+ 0, false, MIGRATE_SYNC);
+ }
+
+ putback_lru_pages(&cc.migratepages);
+ return ret > 0 ? 0 : ret;
+}
+
+/*
+ * Update zone's cma pages counter used for watermark level calculation.
+ */
+static inline void __update_cma_watermarks(struct zone *zone, int count)
+{
+ unsigned long flags;
+ spin_lock_irqsave(&zone->lock, flags);
+ zone->min_cma_pages += count;
+ spin_unlock_irqrestore(&zone->lock, flags);
+ setup_per_zone_wmarks();
+}
+
+/*
+ * Trigger memory pressure bump to reclaim some pages in order to be able to
+ * allocate 'count' pages in single page units. Does similar work as
+ *__alloc_pages_slowpath() function.
+ */
+static int __reclaim_pages(struct zone *zone, gfp_t gfp_mask, int count)
+{
+ enum zone_type high_zoneidx = gfp_zone(gfp_mask);
+ struct zonelist *zonelist = node_zonelist(0, gfp_mask);
+ int did_some_progress = 0;
+ int order = 1;
+
+ /*
+ * Increase level of watermarks to force kswapd do his job
+ * to stabilise at new watermark level.
+ */
+ __update_cma_watermarks(zone, count);
+
+ /* Obey watermarks as if the page was being allocated */
+ while (!zone_watermark_ok(zone, 0, low_wmark_pages(zone), 0, 0)) {
+ wake_all_kswapd(order, zonelist, high_zoneidx, zone_idx(zone));
+
+ did_some_progress = __perform_reclaim(gfp_mask, order, zonelist,
+ NULL);
+ if (!did_some_progress) {
+ /* Exhausted what can be done so it's blamo time */
+ out_of_memory(zonelist, gfp_mask, order, NULL, false);
+ }
+ }
+
+ /* Restore original watermark levels. */
+ __update_cma_watermarks(zone, -count);
+
+ return count;
+}
+
+/**
+ * alloc_contig_range() -- tries to allocate given range of pages
+ * @start: start PFN to allocate
+ * @end: one-past-the-last PFN to allocate
+ * @migratetype: migratetype of the underlaying pageblocks (either
+ * #MIGRATE_MOVABLE or #MIGRATE_CMA). All pageblocks
+ * in range must have the same migratetype and it must
+ * be either of the two.
+ *
+ * The PFN range does not have to be pageblock or MAX_ORDER_NR_PAGES
+ * aligned, however it's the caller's responsibility to guarantee that
+ * we are the only thread that changes migrate type of pageblocks the
+ * pages fall in.
+ *
+ * The PFN range must belong to a single zone.
+ *
+ * Returns zero on success or negative error code. On success all
+ * pages which PFN is in [start, end) are allocated for the caller and
+ * need to be freed with free_contig_range().
+ */
+int alloc_contig_range(unsigned long start, unsigned long end,
+ unsigned migratetype)
+{
+ struct zone *zone = page_zone(pfn_to_page(start));
+ unsigned long outer_start, outer_end;
+ int ret = 0, order;
+
+ /*
+ * What we do here is we mark all pageblocks in range as
+ * MIGRATE_ISOLATE. Because pageblock and max order pages may
+ * have different sizes, and due to the way page allocator
+ * work, we align the range to biggest of the two pages so
+ * that page allocator won't try to merge buddies from
+ * different pageblocks and change MIGRATE_ISOLATE to some
+ * other migration type.
+ *
+ * Once the pageblocks are marked as MIGRATE_ISOLATE, we
+ * migrate the pages from an unaligned range (ie. pages that
+ * we are interested in). This will put all the pages in
+ * range back to page allocator as MIGRATE_ISOLATE.
+ *
+ * When this is done, we take the pages in range from page
+ * allocator removing them from the buddy system. This way
+ * page allocator will never consider using them.
+ *
+ * This lets us mark the pageblocks back as
+ * MIGRATE_CMA/MIGRATE_MOVABLE so that free pages in the
+ * aligned range but not in the unaligned, original range are
+ * put back to page allocator so that buddy can use them.
+ */
+
+ ret = start_isolate_page_range(pfn_max_align_down(start),
+ pfn_max_align_up(end), migratetype);
+ if (ret)
+ goto done;
+
+ ret = __alloc_contig_migrate_range(start, end);
+ if (ret)
+ goto done;
+
+ /*
+ * Pages from [start, end) are within a MAX_ORDER_NR_PAGES
+ * aligned blocks that are marked as MIGRATE_ISOLATE. What's
+ * more, all pages in [start, end) are free in page allocator.
+ * What we are going to do is to allocate all pages from
+ * [start, end) (that is remove them from page allocator).
+ *
+ * The only problem is that pages at the beginning and at the
+ * end of interesting range may be not aligned with pages that
+ * page allocator holds, ie. they can be part of higher order
+ * pages. Because of this, we reserve the bigger range and
+ * once this is done free the pages we are not interested in.
+ *
+ * We don't have to hold zone->lock here because the pages are
+ * isolated thus they won't get removed from buddy.
+ */
+
+ lru_add_drain_all();
+ drain_all_pages();
+
+ order = 0;
+ outer_start = start;
+ while (!PageBuddy(pfn_to_page(outer_start))) {
+ if (++order >= MAX_ORDER) {
+ ret = -EBUSY;
+ goto done;
+ }
+ outer_start &= ~0UL << order;
+ }
+
+ /* Make sure the range is really isolated. */
+ if (test_pages_isolated(outer_start, end)) {
+ pr_warn("alloc_contig_range test_pages_isolated(%lx, %lx) failed\n",
+ outer_start, end);
+ ret = -EBUSY;
+ goto done;
+ }
+
+ /*
+ * Reclaim enough pages to make sure that contiguous allocation
+ * will not starve the system.
+ */
+ __reclaim_pages(zone, GFP_HIGHUSER_MOVABLE, end-start);
+
+ /* Grab isolated pages from freelists. */
+ outer_end = isolate_freepages_range(outer_start, end);
+ if (!outer_end) {
+ ret = -EBUSY;
+ goto done;
+ }
+
+ /* Free head and tail (if any) */
+ if (start != outer_start)
+ free_contig_range(outer_start, start - outer_start);
+ if (end != outer_end)
+ free_contig_range(end, outer_end - end);
+
+done:
+ undo_isolate_page_range(pfn_max_align_down(start),
+ pfn_max_align_up(end), migratetype);
+ return ret;
+}
+
+void free_contig_range(unsigned long pfn, unsigned nr_pages)
+{
+ for (; nr_pages--; ++pfn)
+ __free_page(pfn_to_page(pfn));
+}
+#endif
+
#ifdef CONFIG_MEMORY_HOTREMOVE
/*
* All pages in the range must be isolated before calling this.
* to be MIGRATE_ISOLATE.
* @start_pfn: The lower PFN of the range to be isolated.
* @end_pfn: The upper PFN of the range to be isolated.
+ * @migratetype: migrate type to set in error recovery.
*
* Making page-allocation-type to be MIGRATE_ISOLATE means free pages in
* the range will never be allocated. Any free pages and pages freed in the
* start_pfn/end_pfn must be aligned to pageblock_order.
* Returns 0 on success and -EBUSY if any part of range cannot be isolated.
*/
-int
-start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn)
+int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
+ unsigned migratetype)
{
unsigned long pfn;
unsigned long undo_pfn;
for (pfn = start_pfn;
pfn < undo_pfn;
pfn += pageblock_nr_pages)
- unset_migratetype_isolate(pfn_to_page(pfn));
+ unset_migratetype_isolate(pfn_to_page(pfn), migratetype);
return -EBUSY;
}
/*
* Make isolated pages available again.
*/
-int
-undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn)
+int undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,
+ unsigned migratetype)
{
unsigned long pfn;
struct page *page;
page = __first_valid_page(pfn, pageblock_nr_pages);
if (!page || get_pageblock_migratetype(page) != MIGRATE_ISOLATE)
continue;
- unset_migratetype_isolate(page);
+ unset_migratetype_isolate(page, migratetype);
}
return 0;
}
* all pages in [start_pfn...end_pfn) must be in the same zone.
* zone->lock must be held before call this.
*
- * Returns 1 if all pages in the range is isolated.
+ * Returns 1 if all pages in the range are isolated.
*/
static int
__test_page_isolated_in_pageblock(unsigned long pfn, unsigned long end_pfn)
struct vm_struct *vmlist;
static void setup_vmalloc_vm(struct vm_struct *vm, struct vmap_area *va,
- unsigned long flags, void *caller)
+ unsigned long flags, const void *caller)
{
vm->flags = flags;
vm->addr = (void *)va->va_start;
}
static void insert_vmalloc_vm(struct vm_struct *vm, struct vmap_area *va,
- unsigned long flags, void *caller)
+ unsigned long flags, const void *caller)
{
setup_vmalloc_vm(vm, va, flags, caller);
insert_vmalloc_vmlist(vm);
static struct vm_struct *__get_vm_area_node(unsigned long size,
unsigned long align, unsigned long flags, unsigned long start,
- unsigned long end, int node, gfp_t gfp_mask, void *caller)
+ unsigned long end, int node, gfp_t gfp_mask, const void *caller)
{
struct vmap_area *va;
struct vm_struct *area;
struct vm_struct *__get_vm_area_caller(unsigned long size, unsigned long flags,
unsigned long start, unsigned long end,
- void *caller)
+ const void *caller)
{
return __get_vm_area_node(size, 1, flags, start, end, -1, GFP_KERNEL,
caller);
}
struct vm_struct *get_vm_area_caller(unsigned long size, unsigned long flags,
- void *caller)
+ const void *caller)
{
return __get_vm_area_node(size, 1, flags, VMALLOC_START, VMALLOC_END,
-1, GFP_KERNEL, caller);
}
-static struct vm_struct *find_vm_area(const void *addr)
+/**
+ * find_vm_area - find a continuous kernel virtual area
+ * @addr: base address
+ *
+ * Search for the kernel VM area starting at @addr, and return it.
+ * It is up to the caller to do all required locking to keep the returned
+ * pointer valid.
+ */
+struct vm_struct *find_vm_area(const void *addr)
{
struct vmap_area *va;
static void *__vmalloc_node(unsigned long size, unsigned long align,
gfp_t gfp_mask, pgprot_t prot,
- int node, void *caller);
+ int node, const void *caller);
static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,
- pgprot_t prot, int node, void *caller)
+ pgprot_t prot, int node, const void *caller)
{
const int order = 0;
struct page **pages;
*/
void *__vmalloc_node_range(unsigned long size, unsigned long align,
unsigned long start, unsigned long end, gfp_t gfp_mask,
- pgprot_t prot, int node, void *caller)
+ pgprot_t prot, int node, const void *caller)
{
struct vm_struct *area;
void *addr;
*/
static void *__vmalloc_node(unsigned long size, unsigned long align,
gfp_t gfp_mask, pgprot_t prot,
- int node, void *caller)
+ int node, const void *caller)
{
return __vmalloc_node_range(size, align, VMALLOC_START, VMALLOC_END,
gfp_mask, prot, node, caller);
if (v->flags & VM_IOREMAP)
seq_printf(m, " ioremap");
+ if (v->flags & VM_DMA)
+ seq_printf(m, " dma");
+
if (v->flags & VM_ALLOC)
seq_printf(m, " vmalloc");
"Reclaimable",
"Movable",
"Reserve",
+#ifdef CONFIG_CMA
+ "CMA",
+#endif
"Isolate",
};
config SND_SOC_SAMSUNG
tristate "ASoC support for Samsung"
- depends on ARCH_S3C24XX || ARCH_S3C64XX || ARCH_S5PC100 || ARCH_S5PV210 || ARCH_S5P64X0 || ARCH_EXYNOS4
+ depends on PLAT_SAMSUNG
select S3C64XX_DMA if ARCH_S3C64XX
select S3C2410_DMA if ARCH_S3C24XX
help
config SND_SOC_SAMSUNG_SMDK_WM8994
tristate "SoC I2S Audio support for WM8994 on SMDK"
- depends on SND_SOC_SAMSUNG && (MACH_SMDKV310 || MACH_SMDKC210 || MACH_SMDK4212)
+ depends on SND_SOC_SAMSUNG && (MACH_SMDKV310 || MACH_SMDKC210 || MACH_SMDK4212 || MACH_SMDK4412 || SOC_EXYNOS5250)
depends on I2C=y && GENERIC_HARDIRQS
select MFD_WM8994
select SND_SOC_WM8994
config SND_SOC_SAMSUNG_SMDK_SPDIF
tristate "SoC S/PDIF Audio support for SMDK"
- depends on SND_SOC_SAMSUNG && (MACH_SMDKC100 || MACH_SMDKC110 || MACH_SMDKV210 || MACH_SMDKV310 || MACH_SMDK4212)
+ depends on SND_SOC_SAMSUNG && (MACH_SMDKC100 || MACH_SMDKC110 || MACH_SMDKV210 || MACH_SMDKV310 || MACH_SMDK4212 || MACH_SMDK4412 || SOC_EXYNOS5250)
select SND_SAMSUNG_SPDIF
help
Say Y if you want to add support for SoC S/PDIF audio on the SMDK.
config SND_SOC_SMDK_WM8994_PCM
tristate "SoC PCM Audio support for WM8994 on SMDK"
- depends on SND_SOC_SAMSUNG && (MACH_SMDKC210 || MACH_SMDKV310 || MACH_SMDK4212)
+ depends on SND_SOC_SAMSUNG && (MACH_SMDKC210 || MACH_SMDKV310 || MACH_SMDK4212 || MACH_SMDK4412 || SOC_EXYNOS5250)
depends on I2C=y && GENERIC_HARDIRQS
select MFD_WM8994
select SND_SOC_WM8994
? DMA_MEM_TO_DEV : DMA_DEV_TO_MEM);
dma_info.width = prtd->params->dma_size;
dma_info.fifo = prtd->params->dma_addr;
+ dma_info.dt_dmach_prop = prtd->params->dma_prop;
prtd->params->ch = prtd->params->ops->request(
prtd->params->channel, &dma_info);
}
int dma_size; /* Size of the DMA transfer */
unsigned ch;
struct samsung_dma_ops *ops;
+ struct property *dma_prop;
};
#endif
#include <linux/clk.h>
#include <linux/io.h>
#include <linux/module.h>
+#include <linux/of.h>
#include <linux/pm_runtime.h>
#include <sound/soc.h>
#include <sound/pcm_params.h>
#include <plat/audio.h>
+#include <plat/dma-pl330.h>
#include "dma.h"
#include "idma.h"
struct i2s_dai *i2s_alloc_dai(struct platform_device *pdev, bool sec)
{
struct i2s_dai *i2s;
+ int id;
i2s = devm_kzalloc(&pdev->dev, sizeof(struct i2s_dai), GFP_KERNEL);
if (i2s == NULL)
i2s->i2s_dai_drv.capture.rates = SAMSUNG_I2S_RATES;
i2s->i2s_dai_drv.capture.formats = SAMSUNG_I2S_FMTS;
} else { /* Create a new platform_device for Secondary */
+ if (pdev->dev.of_node) {
+ id = of_alias_get_id(pdev->dev.of_node, "i2s");
+ if (id < 0)
+ dev_err(&pdev->dev,
+ "failed to get alias id:%d\n", id);
+ } else {
+ id = pdev->id;
+ }
+
i2s->pdev = platform_device_register_resndata(NULL,
- pdev->name, pdev->id + SAMSUNG_I2S_SECOFF,
+ "samsung-i2s", id + SAMSUNG_I2S_SECOFF,
NULL, 0, NULL, 0);
if (IS_ERR(i2s->pdev))
return NULL;
static __devinit int samsung_i2s_probe(struct platform_device *pdev)
{
- u32 dma_pl_chan, dma_cp_chan, dma_pl_sec_chan;
+ u32 dma_pl_chan, dma_cp_chan;
+ u32 dma_pl_sec_chan = 0;
struct i2s_dai *pri_dai, *sec_dai = NULL;
struct s3c_audio_pdata *i2s_pdata;
struct samsung_i2s *i2s_cfg;
struct resource *res;
u32 regs_base, quirks;
- int ret = 0;
+ struct property *prop;
+ int ret = 0, id;
/* Call during Seconday interface registration */
- if (pdev->id >= SAMSUNG_I2S_SECOFF) {
+ if (pdev->dev.of_node) {
+ id = of_alias_get_id(pdev->dev.of_node, "i2s");
+ if (id < 0)
+ dev_err(&pdev->dev, "failed to get alias id:%d\n", id);
+ } else {
+ id = pdev->id;
+ }
+
+ if (id >= SAMSUNG_I2S_SECOFF) {
sec_dai = dev_get_drvdata(&pdev->dev);
snd_soc_register_dai(&sec_dai->pdev->dev,
&sec_dai->i2s_dai_drv);
return -EINVAL;
}
- res = platform_get_resource(pdev, IORESOURCE_DMA, 0);
- if (!res) {
- dev_err(&pdev->dev, "Unable to get I2S-TX dma resource\n");
- return -ENXIO;
- }
- dma_pl_chan = res->start;
-
- res = platform_get_resource(pdev, IORESOURCE_DMA, 1);
- if (!res) {
- dev_err(&pdev->dev, "Unable to get I2S-RX dma resource\n");
- return -ENXIO;
- }
- dma_cp_chan = res->start;
-
- res = platform_get_resource(pdev, IORESOURCE_DMA, 2);
- if (res)
- dma_pl_sec_chan = res->start;
- else
- dma_pl_sec_chan = 0;
-
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
if (!res) {
dev_err(&pdev->dev, "Unable to get I2S SFR address\n");
goto err;
}
+ if (!pdev->dev.of_node) {
+ res = platform_get_resource(pdev, IORESOURCE_DMA, 0);
+ if (!res) {
+ dev_err(&pdev->dev,
+ "Unable to get I2S-TX dma resource\n");
+ return -ENXIO;
+ }
+ dma_pl_chan = res->start;
+
+ res = platform_get_resource(pdev, IORESOURCE_DMA, 1);
+ if (!res) {
+ dev_err(&pdev->dev,
+ "Unable to get I2S-RX dma resource\n");
+ return -ENXIO;
+ }
+ dma_cp_chan = res->start;
+
+ res = platform_get_resource(pdev, IORESOURCE_DMA, 2);
+ if (res)
+ dma_pl_sec_chan = res->start;
+ } else {
+ prop = of_find_property(pdev->dev.of_node,
+ "tx-dma-channel", NULL);
+ dma_pl_chan = DMACH_DT_PROP;
+ pri_dai->dma_playback.dma_prop = prop;
+
+ prop = of_find_property(pdev->dev.of_node,
+ "rx-dma-channel", NULL);
+ dma_cp_chan = DMACH_DT_PROP;
+ pri_dai->dma_capture.dma_prop = prop;
+ }
+
pri_dai->dma_playback.dma_addr = regs_base + I2STXD;
pri_dai->dma_capture.dma_addr = regs_base + I2SRXD;
pri_dai->dma_playback.client =
sec_dai->dma_playback.dma_addr = regs_base + I2STXDS;
sec_dai->dma_playback.client =
(struct s3c2410_dma_client *)&sec_dai->dma_playback;
+
+ if (pdev->dev.of_node) {
+ prop = of_find_property(pdev->dev.of_node,
+ "tx-dma-channel-secondary",
+ NULL);
+ sec_dai->dma_playback.dma_prop = prop;
+ }
+
/* Use iDMA always if SysDMA not provided */
sec_dai->dma_playback.channel = dma_pl_sec_chan ? : -1;
sec_dai->src_clk = i2s_cfg->src_clk;
return 0;
}
+#ifdef CONFIG_OF
+static const struct of_device_id exynos_i2s_match[] = {
+ { .compatible = "samsung,i2s" },
+ {},
+};
+MODULE_DEVICE_TABLE(of, exynos_i2s_match);
+#endif
+
static struct platform_driver samsung_i2s_driver = {
.probe = samsung_i2s_probe,
.remove = __devexit_p(samsung_i2s_remove),
.driver = {
.name = "samsung-i2s",
.owner = THIS_MODULE,
+ .of_match_table = of_match_ptr(exynos_i2s_match),
},
};
#include <sound/pcm_params.h>
#include <plat/audio.h>
-#include <plat/dma.h>
+#include <mach/dma.h>
#include "dma.h"
#include "pcm.h"
projects / cascardo / linux.git / commitdiff